diff --git a/CHANGELOG.md b/CHANGELOG.md index 3198a68..381e982 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ All notable public changes to `sprite-gen` are recorded here. Versions track the `version:` field in `SKILL.md` and `pyproject.toml`. -## Unreleased (v2.5.0) +## v2.5.0 - API image generation and smoother loop cuts - New `openai` image provider: `sprite-gen gen --provider openai` calls the OpenAI Images REST API with nothing but `OPENAI_API_KEY` — the credential a headless container (a Modal worker, CI, a SaaS backend) can have, where the `codex` route's interactive ChatGPT login cannot exist. New images go to `/v1/images/generations`, `--ref` switches to `/v1/images/edits` as multipart with the references as repeated `image[]` parts in order (up to 16), and gpt-image's inline base64 is decoded and published as a verified PNG without resizing. Default model `gpt-image-2.5-flare`. `--transparent` asks for `background: transparent` with `output_format: png` — the same measured `native` strategy as codex, and a live run came back 84 % alpha-0 with the subject at alpha 251–254. `--aspect-ratio` maps to one of the gpt-image `size` values that satisfy the API's constraints (both sides divisible by 16, ratio within 1:3..3:1, 655,360–8,294,400 pixels); a ratio with no exact size is refused rather than rounded to a nearby one you would be billed for. A missing or empty key, a rejected key and a failed request are all terminal: this provider never falls back to codex, to another credential, or to a retry. - sprite-gen stays subscription-first. `codex` and `grok` run on a subscription you already pay for; `openai` bills per call, so it runs **only** when `--provider openai` names it. `SPRITE_GEN_DEFAULT_PROVIDER=openai` is refused rather than honoured, the guided `workflow` flow neither offers it nor saves it as a preference, no availability fallback targets it (a codex outage still reaches grok, never metered credit), and having `OPENAI_API_KEY` in the environment changes no route by itself. Every call that does spend API credit — image or video — prints one stderr line naming the charge before the request leaves. @@ -10,6 +10,12 @@ All notable public changes to `sprite-gen` are recorded here. Versions track the - `video`, `video-extend` and `video-edit` announce the per-call charge on stderr when they run on `XAI_API_KEY` instead of the Grok subscription login — one line per submitted job, naming the duration and resolution it is priced on. The subscription route stays silent and no other video behaviour changed. - Registering a provider now fails loudly when the workflow catalog has no label for it. The guided flow built its provider choices by zipping labels onto the provider tuple positionally, so a third provider was dropped without a word; labels are a checked mapping (a missing one fails at import) and the flow offers the subscription routes only. `workflow` access probing knows `openai` — ready when the key is set, with `billing: api-credit` and the billing confirmation that goes with it. +- `video-loop` automatic cuts now score the transition actually played from the last displayed frame back to the first. The chooser prefers a step comparable to the cycle's ordinary motion rather than the smallest distance, which could repeat a pose and pause at the wrap. Reports retain `next_frame_distance` as a separate diagnostic. +- `video-frames --spill full` now lowers the tint threshold as well as lifting the cluster-size limit, so faint key-colour reflections are corrected. `--spill auto` examines the reference at the same threshold before selecting full correction, preserving the character's own key-coloured material. The conservative `small` mode and the default 3-second video-set duration are unchanged. + +- Gait auto-detection now considers an ambiguous double-period candidate even when the short candidate exceeds the duration floor. It preserves both phase occurrences at source speed when the longer local minimum is similarly credible and passes the periodicity gate; near-exact short repeats, worse doubles, non-gait states and caller window bounds are preserved. Reports identify this conservative choice with `half_period_guard.reason = "ambiguous-harmonic"` and `cycle.review_recommended`, rather than claiming anatomical stride certainty. +- Added a [loop review guide](docs/loop-review.md) defining what numeric gates establish, when sequential visual review is needed, and how to record explicit cut/alignment choices without presenting them as automatic decisions. + ## v2.4.1 - A visible hop is no longer refused as "never leaves its rest pose" - `video-loop` one-shot detection no longer refuses a visible hop because the rest pose is not one pose. The excursion is admitted by either the peak's height in MADs of the rest noise (as before) or the fraction of the subject's pixel mass the peak moves (new, ≥ 0.4); the report records `excursion_moved` and `excursion_rule`. A body that walked a few steps, hopped 42 px and froze scored 1.8 MADs because the walking preamble and the frozen tail inflate the "rest noise"; by moved mass it scores 1.07. A jittering stand still fails both rules, and clips the MAD rule already accepted are cut exactly as before. diff --git a/SKILL.md b/SKILL.md index 64d6aad..261eb2d 100644 --- a/SKILL.md +++ b/SKILL.md @@ -1,6 +1,6 @@ --- name: sprite-gen -version: 2.4.1 +version: 2.5.0 description: "Generates images and game sprites through GPT or Grok with guided provider choices, separate saved defaults, automatic cleanup and optional curation. Handles sprite requests, ordinary image generation/editing, standalone image-to-video clips (i2v, animate this still, 그록 영상, 이매진 비디오, 스틸 움직여줘, first/last frame, reference-to-video, 영상 이어붙이기, 영상 편집, extend/edit a clip), chroma removal, animation atlases, video loops, 큐레이션뷰, image candidates, 팔레트 스왑, palette swap, recolor, rig layers, engine exports, repeating backgrounds, projected shadows, motion/contact inspection and optional scene composition from existing assets." license: Apache-2.0 depends_on: diff --git a/docs/README.md b/docs/README.md index 89e954a..0ffa212 100644 --- a/docs/README.md +++ b/docs/README.md @@ -76,6 +76,7 @@ grouping is derived from `sprite_gen/_modules.py`, the one taxonomy table. | [gen.md](gen.md) | `sprite-gen gen` / `gen-set`: providers, default resolution, transparency strategy per provider, row usage | | [video.md](video.md) | `sprite-gen video` / `video-extend` / `video-edit`: stills and clips to mp4 through Grok Imagine (image-to-video, last-frame pin, references, extension, editing) with the user's own credential | | [video-pipeline.md](video-pipeline.md) | Pipeline B engine contract: state canvas, keyed frames, true-period and one-shot cycles, strip/GIF/WebP, the batch | +| [loop-review.md](loop-review.md) | Automatic loop decisions, ambiguous gait review, visual evidence and explicit cut/alignment overrides | | [frame-interpolation.md](frame-interpolation.md) | Generative in-betweens for sprite frames, recorded as a take | | [seamless-video-loop.md](seamless-video-loop.md) | Making a non-looping ambient clip loop forever (RIFE seam bridge) — a different job from pipeline B | diff --git a/docs/loop-review.md b/docs/loop-review.md new file mode 100644 index 0000000..3017c7b --- /dev/null +++ b/docs/loop-review.md @@ -0,0 +1,86 @@ +# Loop automation and visual review + +Use the normal pipeline first. No model needs to choose every cut: + +```bash +sprite-gen video-loop --frames-dir keyed/ --out-dir loop/ --state run --fps 24 +``` + +The caller must pass the **measured source fps**, not always 24. `video-set` obtains it +from `video-frames`. The default anchor is `none` in both commands. The default clip +request remains 3 seconds; it does not determine the playback speed. + +## What the script decides + +- It measures repeat candidates from the keyed frames and the state's time window. +- It scores the last displayed frame -> first frame against ordinary adjacent motion, + reducing repeated-pose stalls at a loop boundary. +- Gaits below the duration floor may use a double period, as before. Above that floor, + similarly credible non-exact candidates at P and approximately 2P now retain 2P once + when it fits the window and passes the periodicity gate. Keeping two real repetitions + is preferable to discarding a potentially distinct phase. This may produce a two-cycle + loop of a genuine short gait; frames keep their original timing. +- Near-exact short repeats, implausible longer repeats, and non-gait states are not + extended by that ambiguity policy. There is no character-specific frame count. +- Alpha, frame counts and seam metrics are checked before the output is accepted. + +A numeric pass establishes those facts. It does **not** establish correct foot contact, +anatomical left/right alternation, a believable gait, or that the source was drawn well. +`cycle.review_recommended = true` marks an ambiguous-harmonic choice; `false` means +that this particular heuristic did not flag ambiguity, not that visual quality is certified. +The direct CLI summary preserves that flag and the guard evidence. `video-set` also +propagates them in its item report and marks the table row `OK (review gait)`. + +## Material for a vision-capable reviewer + +Provide the original clip, the generated loop played at source speed for at least three +repeats, the JSON report, and an ordered contact sheet with original frame indices. For +an ambiguous harmonic include both P and 2P candidates using the same size and fps. +A single still cannot establish a gait cycle. If the model cannot consume video, provide +ordered frames spanning at least two candidate cycles, including several frames on both +sides of the wrap. Do not ask it to judge motion from an unordered image collage. + +A human or a vision-capable model should check: + +1. Do the limbs alternate, or does one phase restart before its counterpart appears? +2. At the wrap, does motion stall, snap, or reverse compared with nearby frames? +3. Does the torso translate across the canvas, or are only the limbs/clothing moving? +4. Are deformation or contact errors already present in the source clip? + +Return the evidence frame indices, chosen mode/start/length, anchor, reason and any +uncertainty. Keep that decision separate from the unmodified automatic report. A model +without the required visual input must not claim it reviewed the gait. Ambiguous or +contradictory visual evidence should remain unresolved rather than inventing a precise cut. + +## Explicit adjustments + +If the visible full cycle is clear but the automatic candidate is wrong, make the choice +explicit and keep the report's `kind = fixed`: + +```bash +sprite-gen video-loop --frames-dir keyed/ --out-dir reviewed/ --state run \ + --fps 24 --cycle fixed --start --length +``` + +Do not double a frame count unconditionally or duplicate existing frames. Choose a +continuous interval in the original sequence. `--min-len/--max-len` can instead narrow +an evidenced search interval while leaving the exact cut to the detector. + +Use `--anchor feet` only when the source shows slow in-canvas translation. It removes a +linear horizontal trend; it is not a universal stabilizer, does not pin each planted foot, +and can alter an already stationary subject. Start with the default `none`. Compare with +`feet` on identical frames before selecting it. Normal body bounce, clothing motion and +a model's changing anatomy are not position drift. + +Do not raise `--seam-max` merely to make a failed result pass. A source lacking a complete +usable cycle may require a longer/new clip. Regeneration is a new generation request, +not a hidden fallback inside loop extraction. + +## Reproducibility + +Store the source identity, engine version, exact CLI arguments, selected report and +whether a person/model overrode the automatic choice. Keep the same fps for comparisons. +Render comparison videos from PNG strips or lossless frame outputs, not palette GIFs. +A reviewed example is not evidence that all future inputs will automatically attain its +quality. Consumers can route `review_recommended` to review while retaining the automatic +result as an explicitly labelled candidate. diff --git a/docs/video-pipeline.md b/docs/video-pipeline.md index d6aa64b..dfc749e 100644 --- a/docs/video-pipeline.md +++ b/docs/video-pipeline.md @@ -140,16 +140,17 @@ The still the clip was made from settles it. `video-frames --spill` takes: | Mode | What is corrected | |---|---| | `small` (default) | only small key-tinted clusters — the still pipeline's rule, byte-identical output | -| `full` | every key-tinted cluster, whatever its size (colour only: alpha is unchanged) | -| `auto` | keys `--reference` (the still) with the same matte and counts its strongly key-tinted pixels; a share ≤ 0.5 % means the still has no key-coloured material of its own → `full`, otherwise `small` | +| `full` | key-tinted clusters of any size, including faint tints (colour only: alpha is unchanged) | +| `auto` | keys `--reference` (the still) with the same matte and counts its key-tinted pixels at the same threshold used by `full`; a share ≤ 0.5 % means the still has no key-coloured material of its own → `full`, otherwise `small` | `video-set` passes `--spill auto` with each item's `canvas.png` as the reference (override with `--spill small|full`), so a green-free character loses the reflections while a character that *is* green keeps its colour. The decision and its numbers are recorded in the frames report under `spill`. The correction is the engine's own `despill_color` blend model (observed = (1−k)·subject + k·key, solved for the subject), so colours without key -tint are untouched. Faint tints whose cluster never crosses the strong-tint bar are left -as they are. +tint are untouched. `small` keeps the conservative tint threshold of 40; `full` lowers it to 8. +The `auto` reference check also uses 8, so faint key-coloured material in the +original character keeps the conservative correction. ## 3b. Canvas shape for raised limbs and wide costumes @@ -173,7 +174,10 @@ was 17). `video-loop` therefore: *smallest* local minimum that is within 15 % of the deepest one — exact repeats dip again at 2× and 3× the period, the half-period look-alike dips noticeably less; 3. only then picks the **start** with the best seam for that period (± 1 frame): - `seam = D[i][i+L]` over the mean adjacent distance inside the cycle. + `seam = D[i+L-1][i]` over the mean adjacent distance inside the cycle. + Choose the ratio closest to 1 in log space, so a repeated pose at the wrap does + not win just because its distance is small. `next_frame_distance` retains the + distance to the frame after the cycle for diagnostics. Windows come from the state profile (`STATE_PROFILES`). **Gait states take theirs in seconds**, because a stride is a fact about the body, not about the clip length: walk @@ -192,6 +196,16 @@ periodicity gate measures the period's dip against the profile mean over the who window, and a ceiling that grows with the clip inflates that mean until a single hop in a jittering stand passes as a walk. +For gait states, a duration above the floor is not proof that both phases are present. +If a local minimum near twice the chosen period is within the existing 25 % repeat-error +tolerance and still passes the periodicity gate, the detector retains the longer candidate +once, inside the requested window. Near-exact repeats (repeat error at most 10 % of an +ordinary adjacent step) stay short. This is a conservative response to ambiguous harmonics: +a genuine short gait may be shown twice, at the same source speed. It does not identify +anatomical left/right contacts. The report records `half_period_guard.reason = +"ambiguous-harmonic"` and `cycle.review_recommended = true`. See [loop review](loop-review.md) +for the visual review contract and manual overrides. + Gates, all fail-loud: no period (profile flat, `periodicity < 0.15`), loop seam ratio above `--seam-max` (2.0), GIF/WebP re-opened and checked (frame count, `loop=0`, transparent corners, no RGB under alpha 0 in the WebP). diff --git a/pyproject.toml b/pyproject.toml index 23cd3a0..9d183cd 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -12,7 +12,7 @@ build-backend = "setuptools.build_meta" name = "sprite-gen" # Release discipline: keep this package metadata version synchronized with # SKILL.md's `version:` field in the same release commit. -version = "2.4.1" +version = "2.5.0" description = "Component-row pipeline for clean 2D game sprites and animation atlases" readme = "README.md" license = "Apache-2.0" diff --git a/sprite_gen/frames/cutout.py b/sprite_gen/frames/cutout.py index 452a372..978147a 100644 --- a/sprite_gen/frames/cutout.py +++ b/sprite_gen/frames/cutout.py @@ -271,7 +271,8 @@ def _matte_route( } -def extract_route(image: Image.Image, kind: str, *, spill_max_fraction: float | None = None) -> tuple[Image.Image, dict[str, Any]]: +def extract_route(image: Image.Image, kind: str, *, spill_max_fraction: float | None = None, + spill_min_tint: float | None = None) -> tuple[Image.Image, dict[str, Any]]: """Magenta/green key background → reuse the verified `extract` chroma engine (no drift). The engine keys from the background colour it detects on the borders @@ -285,7 +286,11 @@ def extract_route(image: Image.Image, kind: str, *, spill_max_fraction: float | target = KEY_TARGETS[kind] painted = detect_background_key_rgb(image, target) - extra = {} if spill_max_fraction is None else {"spill_max_fraction": spill_max_fraction} + extra: dict[str, float] = {} + if spill_max_fraction is not None: + extra["spill_max_fraction"] = spill_max_fraction + if spill_min_tint is not None: + extra["spill_min_tint"] = spill_min_tint result = remove_chroma_background( image, target, _EXTRACT_KEY_THRESHOLD, _EXTRACT_FRINGE_THRESHOLD, _EXTRACT_FRINGE_DELTA, **extra ) @@ -307,12 +312,14 @@ def cutout( tolerance: int = CHROMA_TOLERANCE, white_check_dir: Path | None = None, spill_max_fraction: float | None = None, + spill_min_tint: float | None = None, ) -> dict[str, Any]: """Cut a uniform-background imported image to a clean transparent RGBA PNG. `key`: "auto" (detect from corners) | "white" (matte) | "magenta" | "green" (reuse the extract chroma engine). `spill_max_fraction` overrides the chroma - engine's trapped-spill cluster cap (None = the engine default). Returns a stats dict. Raises SystemExit if + engine's trapped-spill cluster cap and `spill_min_tint` its tint bar (None = the + engine default for either). Returns a stats dict. Raises SystemExit if the key is unknown, the background cannot be located, or any transparent pixel keeps non-zero RGB (No Silent Fallback). """ @@ -324,7 +331,8 @@ def cutout( route = _detect_key_kind(_corner_average(image)) if key == "auto" else key if route in ("magenta", "green"): - result, route_stats = extract_route(image, route, spill_max_fraction=spill_max_fraction) + result, route_stats = extract_route(image, route, spill_max_fraction=spill_max_fraction, + spill_min_tint=spill_min_tint) else: result, route_stats = _matte_route(image, input_path, strength, band, erode, tolerance) diff --git a/sprite_gen/frames/extract.py b/sprite_gen/frames/extract.py index 15de7d0..ef1d324 100644 --- a/sprite_gen/frames/extract.py +++ b/sprite_gen/frames/extract.py @@ -362,19 +362,27 @@ def detect_background_key_rgb( # channel clears every unkeyed channel by >40): warm subject colors (skin) # score a marginal tint just above fringe_delta and must not be "corrected". _SPILL_MIN_TINT = 40.0 +# Full correction also admits faint key tints. Auto mode must inspect the +# reference at this same threshold before choosing full correction. +_SPILL_FULL_MIN_TINT = 8.0 -def key_material_pixels(image: Image.Image, chroma_key: tuple[int, int, int]) -> tuple[int, int]: - """Opaque pixels that are strongly key-tinted (the trapped-spill bar, `_SPILL_MIN_TINT`), - and all opaque pixels, of an already keyed RGBA image. A still that has almost none - carries no key-coloured material of its own.""" +def key_material_pixels(image: Image.Image, chroma_key: tuple[int, int, int], + min_tint: float = _SPILL_MIN_TINT) -> tuple[int, int]: + """Opaque pixels that are key-tinted past `min_tint` (the trapped-spill bar), and all + opaque pixels, of an already keyed RGBA image. A still that has almost none carries no + key-coloured material of its own. + + The bar is a parameter because the decision and the treatment have to read the same + one: judging a reference at 40 and then despilling it down to 8 would call a subject + with a mild green of its own "no key material" and then scrub that green away.""" keyed_channels, unkeyed_channels = _key_channel_split(chroma_key) data = np.asarray(image.convert("RGBA")).astype(np.int32) opaque = data[..., 3] > 0 if not keyed_channels: return 0, int(opaque.sum()) tint = _key_tint_field(data[..., :3], keyed_channels, unkeyed_channels) - return int((opaque & (tint > _SPILL_MIN_TINT)).sum()), int(opaque.sum()) + return int((opaque & (tint > min_tint)).sum()), int(opaque.sum()) def remove_chroma_background( @@ -386,6 +394,7 @@ def remove_chroma_background( *, unmix_reach: int = 4, spill_max_fraction: float = 0.005, + spill_min_tint: float = _SPILL_MIN_TINT, background_key: tuple[int, int, int] | None = None, ) -> Image.Image: """Key `chroma_key` out of `image` (hard cut + soft-alpha fringe unmix + trapped-spill despill). @@ -520,8 +529,13 @@ def remove_chroma_background( # the image, and a pixel it despilled is no longer a spill candidate. current_tint = _key_tint_field(data[..., :3].astype(np.int32), keyed_channels, unkeyed_channels) + # Candidacy and acceptance read the same bar: a cluster can never be accepted + # below `spill_min_tint`, so admitting only pixels at or above `fringe_delta` + # would silently keep the lowered bar from reaching anything when it is the + # smaller of the two. + candidate_tint = min(fringe_delta, spill_min_tint) candidates = np.flatnonzero( - ((data[..., 3] != 0) & (current_tint >= fringe_delta)).reshape(-1) + ((data[..., 3] != 0) & (current_tint >= candidate_tint)).reshape(-1) ) tints_left: dict[int, float] = dict( zip(candidates.tolist(), current_tint.reshape(-1)[candidates].tolist()) @@ -547,7 +561,7 @@ def remove_chroma_background( stack.append(neighbor) if len(cluster) > spill_limit: continue - if max(tints_left[index] for index in cluster) <= _SPILL_MIN_TINT: + if max(tints_left[index] for index in cluster) <= spill_min_tint: continue for index in cluster: x = index % width diff --git a/sprite_gen/video/batch.py b/sprite_gen/video/batch.py index 711bb8e..88c8194 100644 --- a/sprite_gen/video/batch.py +++ b/sprite_gen/video/batch.py @@ -134,6 +134,8 @@ def run_item( result["frames"]["spill"] = fr.get("spill", {}).get("mode") lp = loop_mod.run_loop(Path(fr["keyed_dir"]), item_dir / "loop", fps=float(fr["fps"]), state=state, min_len=None, max_len=None, n_out=None, seam_max=loop_mod.SEAM_RATIO_MAX, name=item, report_path=item_dir / "loop.report.json", anchor=anchor) result["loop"] = {"kind": lp["cycle"].get("kind", "periodic"), "cycle": lp["cycle"]["length"], "period": lp["cycle"]["period_global"], "cycle_ratio": round(lp["cycle"]["ratio"], 3), "seam_ratio": lp["resampled_seam_ratio"], "n_out": lp["n_out"], "drift_px": lp["strip"].get("drift_px", 0), "gif": lp["gif"]["file"], "webp": lp["webp"]["file"], "strip": lp["strip"]["path"]} + result["loop"]["review_recommended"] = lp["cycle"].get("review_recommended", False) + result["loop"]["half_period_guard"] = lp["cycle"].get("half_period_guard") result["ok"] = True except SystemExit as exc: result["ok"] = False @@ -146,7 +148,8 @@ def write_table(results: list[dict[str, Any]], path: Path) -> str: for r in results: if r.get("ok"): lp = r["loop"] - lines.append(f"| {r['direction']} | {r['state']} | {lp.get('kind', 'periodic')} | {lp['cycle']} | {lp['period'] if lp['period'] is not None else '-'} | {lp['seam_ratio']:.2f} | {lp['n_out']} | OK |") + status = "OK (review gait)" if lp.get("review_recommended") else "OK" + lines.append(f"| {r['direction']} | {r['state']} | {lp.get('kind', 'periodic')} | {lp['cycle']} | {lp['period'] if lp['period'] is not None else '-'} | {lp['seam_ratio']:.2f} | {lp['n_out']} | {status} |") else: lines.append(f"| {r['direction']} | {r['state']} | - | - | - | - | - | FAIL: {r.get('error', '')[:80]} |") text = "\n".join(lines) + "\n" diff --git a/sprite_gen/video/frames.py b/sprite_gen/video/frames.py index 5d266f0..4c13174 100644 --- a/sprite_gen/video/frames.py +++ b/sprite_gen/video/frames.py @@ -33,6 +33,7 @@ from sprite_gen.frames.cutout import cutout from sprite_gen.frames.extract import is_border_key_candidate +from sprite_gen.frames.extract import _SPILL_FULL_MIN_TINT from sprite_gen.spec.runio import atomic_write_text EDGE_ROWS = 4 # rows/cols inspected at each edge @@ -44,6 +45,7 @@ # key tint in the clip was painted by the model and is spill. SPILL_MODES = ("auto", "small", "full") SPILL_FULL_FRACTION = 1.0 # every tinted cluster is spill, whatever its size +SPILL_FULL_MIN_TINT = _SPILL_FULL_MIN_TINT # ... and whatever its strength (see extract.py) SPILL_REFERENCE_MAX = 0.005 # the still's own key material ≤ the engine's small-cluster share → full EDGE_MAX_PIXELS = 0 # any opaque pixel on the top/left/right edge band = contact @@ -135,7 +137,9 @@ def decide_spill(reference: Path, key: str) -> dict[str, Any]: if kind not in ("green", "magenta"): return {"mode": "small", "reference": str(reference), "reason": f"key {kind!r} has no spill pass"} keyed, _ = extract_route(image, kind) - material, subject = key_material_pixels(keyed, KEY_TARGETS[kind]) + # judged at the bar `full` would treat with, so a subject that owns a mild key tint + # is not first called "no key material" and then scrubbed of it + material, subject = key_material_pixels(keyed, KEY_TARGETS[kind], SPILL_FULL_MIN_TINT) share = material / subject if subject else 0.0 mode = "full" if share <= SPILL_REFERENCE_MAX else "small" return {"mode": mode, "reference": str(reference), "key": kind, "key_material_px": material, @@ -152,13 +156,15 @@ def key_frames( ) -> dict[str, Any]: if spill not in ("small", "full"): raise SystemExit(f"video-frames: key_frames takes a resolved spill mode (small|full), got {spill!r}") + # Full correction lowers the tint threshold as well as lifting the size cap. spill_max = SPILL_FULL_FRACTION if spill == "full" else None + spill_tint = SPILL_FULL_MIN_TINT if spill == "full" else None keyed_dir.mkdir(parents=True, exist_ok=True) rows: list[dict[str, Any]] = [] contacts: list[dict[str, Any]] = [] for src in raw_files: dst = keyed_dir / src.name - stats = cutout(src, dst, key=key, spill_max_fraction=spill_max) + stats = cutout(src, dst, key=key, spill_max_fraction=spill_max, spill_min_tint=spill_tint) image = Image.open(dst).convert("RGBA") hist = image.getchannel("A").histogram() w, h = image.size diff --git a/sprite_gen/video/loop.py b/sprite_gen/video/loop.py index f0acc2e..14ea31e 100644 --- a/sprite_gen/video/loop.py +++ b/sprite_gen/video/loop.py @@ -69,6 +69,7 @@ # taken when it repeats about as well. The cost is asymmetric — a wrongly doubled cycle # is still a clean two-cycle loop, a halved one walks on one leg. GAIT_DOUBLE_TOL = 0.25 +GAIT_NEAR_EXACT_STEP_FRACTION = 0.10 # no ambiguity extension when repeat error is tiny compared with a playback step ANCHOR_MODES = ("none", "feet") FOOT_BAND = 0.08 # fraction of the frame's own height, measured up from its lowest opaque row @@ -157,7 +158,8 @@ def detect_cycle(D: np.ndarray, *, min_len: int, max_len: int, gait_floor: int | `gait_floor` (frames) turns on the half-period guard: a period below it is one step of a two-step gait, so the doubled period is taken when it repeats about as well - (see GAIT_DOUBLE_TOL).""" + (see GAIT_DOUBLE_TOL). Above the floor, ambiguous non-exact harmonics may also + retain two phase occurrences; the report flags that decision for visual review.""" n = D.shape[0] max_len = min(max_len, n - 2) if min_len < 2 or max_len < min_len: @@ -184,21 +186,54 @@ def detect_cycle(D: np.ndarray, *, min_len: int, max_len: int, gait_floor: int | period = L2 else: guard = {"applied": False, "below_floor": period, "gait_floor": gait_floor, "why": "the doubled period repeats too much worse to be the same gait"} + # A plausible duration does not prove that a gait contains both phases. If a + # second local minimum at twice the period is similarly good, retain both + # occurrences at the original fps. This is a conservative ambiguity policy, + # not an anatomical inference: a true short cycle may be shown twice. + # Do it only once, inside the requested window, and leave near-exact repeats + # alone. The duration-floor guard above still owns implausibly short beats. profile_mean = float(np.mean([prof[L] for L in prof])) + review_recommended = False + if gait_floor is not None and period >= gait_floor and not guard["applied"]: + ordinary_step = float(adjacent.mean()) + repeat_fraction = prof[period] / ordinary_step if ordinary_step > 0 else math.inf + doubles = [L for L in cands if abs(L - 2 * period) <= 1] + if doubles and repeat_fraction > GAIT_NEAR_EXACT_STEP_FRACTION: + L2 = min(doubles, key=lambda L: prof[L]) + if (prof[L2] <= prof[period] * (1 + GAIT_DOUBLE_TOL) + 1e-4 + and profile_mean > 0 + and (profile_mean - prof[L2]) / profile_mean >= PERIODICITY_MIN): + guard = { + "applied": True, "from": period, "to": L2, + "gait_floor": gait_floor, "reason": "ambiguous-harmonic", + "depth_ratio": round(prof[L2] / prof[period], 3), + "repeat_error_over_step": round(repeat_fraction, 3), + "why": "both periods are plausible; retain two phase occurrences at source speed", + } + period = L2 + review_recommended = True periodicity = (profile_mean - prof[period]) / profile_mean if profile_mean > 0 else 0.0 + # Score the last displayed frame -> first frame transition against an ordinary + # playback step. Minimising distance alone rewards a repeated pose (a stall). + # Log distance penalises steps that are too short or too long symmetrically. best: dict[str, Any] | None = None for L in (period - 1, period, period + 1): if L < min_len or L > max_len: continue for i in range(0, n - L): - seam = float(D[i, i + L]) + seam = float(D[i + L - 1, i]) inner = float(adjacent[i : i + L - 1].mean()) ratio = seam / inner if inner > 0 else math.inf - if best is None or ratio < best["ratio"]: - best = {"start": i, "length": L, "seam": seam, "inner_mean_adjacent": inner, "ratio": ratio} + score = abs(math.log(ratio)) if ratio > 0 else math.inf + if best is None or score < best["wrap_score"]: + best = {"start": i, "length": L, "seam": seam, "inner_mean_adjacent": inner, + "ratio": ratio, "wrap_score": score, + "next_frame_distance": float(D[i, i + L]) if i + L < n else None} assert best is not None + best.pop("wrap_score") best["period_global"] = period best["half_period_guard"] = guard + best["review_recommended"] = review_recommended best["periodicity"] = round(periodicity, 4) # how far below the profile mean the period dips (0 = flat = no period) best["profile_minima"] = [[L, round(prof[L], 5)] for L in sorted(cands, key=lambda L: prof[L])[:6]] return best @@ -709,7 +744,7 @@ def run(**kwargs: object) -> int: anchor=str(kwargs.get("anchor") or "none"), ) summary = {k: payload[k] for k in ("state", "frames_total", "window", "cycle_seconds", "n_out", "delay_ms", "resampled_seam_ratio", "specks_dropped", "report")} - summary["cycle"] = {k: payload["cycle"].get(k) for k in ("kind", "start", "length", "period_global", "ratio")} + summary["cycle"] = {k: payload["cycle"].get(k) for k in ("kind", "start", "length", "period_global", "ratio", "review_recommended", "half_period_guard")} if payload["periodic_attempt"]: summary["periodic_attempt"] = payload["periodic_attempt"]["why_rejected"] summary["strip"] = {k: payload["strip"][k] for k in ("path", "frames", "w", "h", "body_h", "delay_ms")} diff --git a/tests/frames/test_despill_full_tint_bar.py b/tests/frames/test_despill_full_tint_bar.py new file mode 100644 index 0000000..76a71ef --- /dev/null +++ b/tests/frames/test_despill_full_tint_bar.py @@ -0,0 +1,82 @@ +"""Full spill correction removes faint tint and preserves reference material.""" + +from __future__ import annotations + +from pathlib import Path + +from PIL import Image + +from sprite_gen._deps import np +from sprite_gen.frames.cutout import KEY_TARGETS, extract_route +from sprite_gen.frames.extract import _SPILL_FULL_MIN_TINT, _SPILL_MIN_TINT, key_material_pixels +from sprite_gen.video import frames as frames_mod + + +def _green_canvas_with_tinted_subject(path: Path, *, tint: int, share: float = 1.0) -> Path: + """A pale subject on a green field. `share` of the subject's area carries `tint` + added to its green channel — the key light bouncing off it, which is what a video + model can paint onto a neutral subject. + """ + key = KEY_TARGETS["green"] + size, box = 160, (32, 32, 128, 128) # a 96x96 subject: 9,216 px, so 1 px is 0.01 % + im = Image.new("RGB", (size, size), key) + left, top, right, bottom = box + area = (right - left) * (bottom - top) + lit = max(1, round(area * share)) + painted = 0 + for y in range(top, bottom): + for x in range(left, right): + on = painted < lit + im.putpixel((x, y), (230, min(255, 225 + (tint if on else 0)), 225)) + painted += on + im.save(path) + return path + + +def _residual_tint(image: Image.Image) -> np.ndarray: + data = np.asarray(image.convert("RGBA")).astype(np.int32) + opaque = data[..., 3] > 0 + tint = data[..., 1] - (data[..., 0] + data[..., 2]) / 2.0 + return tint[opaque] + + +def test_the_full_bar_reaches_a_tint_the_default_bar_leaves_alone(tmp_path: Path) -> None: + src = _green_canvas_with_tinted_subject(tmp_path / "in.png", tint=20) # between the bars + image = Image.open(src).convert("RGBA") + + default_keyed, _ = extract_route(image, "green", spill_max_fraction=1.0) + full_keyed, _ = extract_route(image, "green", spill_max_fraction=1.0, + spill_min_tint=_SPILL_FULL_MIN_TINT) + + assert _SPILL_FULL_MIN_TINT < 20 < _SPILL_MIN_TINT + assert _residual_tint(default_keyed).max() > 15, "the default bar is above this tint" + assert _residual_tint(full_keyed).max() < 15, "the full bar has to reach it" + + +def test_lifting_only_the_size_cap_is_not_enough(tmp_path: Path) -> None: + """The bug as it stood: `full` moved `spill_max_fraction` and nothing changed.""" + src = _green_canvas_with_tinted_subject(tmp_path / "in.png", tint=20) + image = Image.open(src).convert("RGBA") + small, _ = extract_route(image, "green") + size_cap_only, _ = extract_route(image, "green", spill_max_fraction=1.0) + assert _residual_tint(small).max() == _residual_tint(size_cap_only).max() + + +def test_a_subject_that_owns_key_material_is_still_judged_small(tmp_path: Path) -> None: + """The decision reads the bar the treatment will use, so a genuinely green subject + keeps its green: it is `small`, and `full`'s lowered bar never touches it.""" + # a subject that is green material: most of it carries the key hue + owns_green = _green_canvas_with_tinted_subject(tmp_path / "green-subject.png", tint=90) + # a clean subject carrying only what the key bounced onto its lower edge + clean = _green_canvas_with_tinted_subject(tmp_path / "pale-subject.png", tint=20, share=0.003) + assert frames_mod.decide_spill(owns_green, "green")["mode"] == "small" + assert frames_mod.decide_spill(clean, "green")["mode"] == "full" + + +def test_key_material_is_measured_at_the_bar_it_is_given(tmp_path: Path) -> None: + src = _green_canvas_with_tinted_subject(tmp_path / "in.png", tint=20) + keyed, _ = extract_route(Image.open(src).convert("RGBA"), "green") + at_default, subject = key_material_pixels(keyed, KEY_TARGETS["green"]) + at_full, subject_again = key_material_pixels(keyed, KEY_TARGETS["green"], _SPILL_FULL_MIN_TINT) + assert subject == subject_again + assert at_default < at_full, "a lower bar must see at least as much key material" diff --git a/tests/video/test_gait_harmonic_ambiguity.py b/tests/video/test_gait_harmonic_ambiguity.py new file mode 100644 index 0000000..010ed7f --- /dev/null +++ b/tests/video/test_gait_harmonic_ambiguity.py @@ -0,0 +1,56 @@ +"""Ambiguous gaits retain both phases without a fixed cut or a slower frame rate.""" +from __future__ import annotations + +import numpy as np +import pytest + +from sprite_gen.video import loop + + +def _repeat_distances(period=13, *, noise=0.0, drift=0.0): + t = np.arange(period * 12) + angle = t * 2 * np.pi / period + points = np.column_stack([np.cos(angle), np.sin(angle), drift * t]) + rng = np.random.default_rng(31) + nuisance = rng.normal(0, noise, (len(t), 12)) + points = np.column_stack([points, nuisance]) + return np.abs(points[:, None] - points[None, :]).mean(axis=2).astype(np.float32) + + +def test_noisy_harmonic_above_duration_floor_retains_both_occurrences(): + D = _repeat_distances(noise=0.02) + plain = loop.detect_cycle(D, min_len=7, max_len=29) + gait = loop.detect_cycle(D, min_len=7, max_len=29, gait_floor=8) + assert plain['period_global'] == 13 + assert gait['period_global'] == 26 + assert gait['length'] in (25, 26, 27) + assert gait['half_period_guard']['reason'] == 'ambiguous-harmonic' + assert gait['half_period_guard']['from'] == 13 + assert gait['review_recommended'] is True + + +@pytest.mark.parametrize('noise', [0, 0.0001]) +def test_exact_or_near_exact_fast_repeat_is_not_doubled(noise): + cycle = loop.detect_cycle(_repeat_distances(noise=noise), min_len=7, max_len=29, gait_floor=8) + assert cycle['period_global'] == 13 + assert cycle['half_period_guard']['applied'] is False + assert cycle['review_recommended'] is False + + +def test_ambiguity_policy_never_expands_the_callers_window(): + cycle = loop.detect_cycle(_repeat_distances(noise=0.02), min_len=7, max_len=20, gait_floor=8) + assert cycle['period_global'] == 13 + assert cycle['length'] <= 20 + assert cycle['half_period_guard']['applied'] is False + + +def test_worsening_repeat_does_not_trigger_harmonic_extension(): + cycle = loop.detect_cycle(_repeat_distances(noise=0.02, drift=0.08), min_len=7, max_len=29, gait_floor=8) + assert cycle['period_global'] == 13 + assert cycle['half_period_guard']['applied'] is False + + +def test_non_gait_keeps_the_shortest_credible_repeat(): + cycle = loop.detect_cycle(_repeat_distances(noise=0.02), min_len=7, max_len=29) + assert cycle['period_global'] == 13 + assert cycle['review_recommended'] is False diff --git a/tests/video/test_loop_sampling_and_wrap.py b/tests/video/test_loop_sampling_and_wrap.py new file mode 100644 index 0000000..0b1ede2 --- /dev/null +++ b/tests/video/test_loop_sampling_and_wrap.py @@ -0,0 +1,75 @@ +"""Loop cuts preserve an ordinary playback step at the wrap.""" + +from __future__ import annotations + +import math +from pathlib import Path + +import pytest +from PIL import Image + +from sprite_gen._deps import np +from sprite_gen.video import loop as loop_mod + + +def _walker(tmp_path: Path, *, period: int, n: int, size=(64, 64)) -> list[Path]: + """A walker whose stride is `period` frames, with a near/far marker so the half + period is distinguishable — the same body in a 1 s, 2 s or 3 s clip.""" + d = tmp_path / "keyed" + d.mkdir(parents=True, exist_ok=True) + files = [] + for t in range(n): + im = Image.new("RGBA", size, (0, 0, 0, 0)) + for y in range(10, 40): + for x in range(26, 38): + im.putpixel((x, y), (200, 60, 60, 255)) + phase = 2 * math.pi * t / period + leg_x = 32 + round(10 * math.sin(phase)) + for y in range(40, 58): + for x in range(leg_x - 3, leg_x + 3): + if 0 <= x < size[0]: + im.putpixel((x, y), (60, 60, 200, 255)) + if math.sin(phase) >= 0: + im.putpixel((leg_x, 50), (255, 255, 0, 255)) + f = d / f"f{t:03d}.png" + im.save(f) + files.append(f) + return files + + +def test_the_cut_is_scored_on_the_step_that_plays_not_the_frame_after_it(tmp_path: Path) -> None: + """A non-integer period is where the two metrics disagree, and the wrap step is the + one a viewer sees: too small means the last frame repeats the first and the loop + freezes for a frame.""" + files = _walker(tmp_path, period=12, n=96) + D = loop_mod.distance_matrix(files) + cycle = loop_mod.detect_cycle(D, min_len=4, max_len=40) + i, L = cycle["start"], cycle["length"] + adjacent = np.array([D[k, k + 1] for k in range(len(files) - 1)]) + inner = float(adjacent[i : i + L - 1].mean()) + # `seam`/`ratio` report D[i+L-1, i] — the transition from the last shown frame back + # to the first — not D[i, i+L]. + assert cycle["seam"] == pytest.approx(float(D[i + L - 1, i]), rel=1e-9) + assert cycle["ratio"] == pytest.approx(cycle["seam"] / inner, rel=1e-9) + # and it is a real step of the cycle, never a stall + assert cycle["ratio"] > 0.5 + + +def test_a_stall_at_the_wrap_loses_to_an_ordinary_step() -> None: + """A non-integer period is exactly where the two metrics disagree. + + Frames walk around a circle with a synthetic 24.5-frame period. L=25 puts the last shown frame half a step from the + first (wrap 0.50x: the loop freezes for a frame); L=24 plays an ordinary step. + """ + period, n = 24.5, 60 + angle = np.arange(n) * 2 * math.pi / period + points = np.stack([np.cos(angle), np.sin(angle)], axis=1) + D = np.linalg.norm(points[:, None, :] - points[None, :, :], axis=-1).astype(np.float32) + adjacent = float(np.mean([D[k, k + 1] for k in range(n - 1)])) + # the situation, stated: L=25 stalls, L=24 does not + assert D[24, 0] / adjacent == pytest.approx(0.50, abs=0.05) + assert D[23, 0] / adjacent == pytest.approx(1.49, abs=0.05) + + cycle = loop_mod.detect_cycle(D, min_len=20, max_len=28) + assert cycle["length"] == 24 + assert cycle["ratio"] > 0.75, "a cut that repeats its first frame at the wrap must lose" diff --git a/tests/video/test_video_pipeline.py b/tests/video/test_video_pipeline.py index 4eacc3b..98438ad 100644 --- a/tests/video/test_video_pipeline.py +++ b/tests/video/test_video_pipeline.py @@ -170,7 +170,13 @@ def test_detect_cycle_finds_the_full_period_not_the_half_or_1_5x(tmp_path: Path) cycle = loop_mod.detect_cycle(D, min_len=4, max_len=40) assert cycle["period_global"] == 12 assert cycle["length"] in (11, 12, 13) - assert cycle["ratio"] < 1.0 + # The cut closes exactly: the frame that would come next IS the frame it jumps back to. + assert cycle["next_frame_distance"] == pytest.approx(0.0, abs=1e-6) + # `ratio` scores the step that actually plays at the wrap, so it is a step of this + # cycle and not a number that shrinks toward zero. The fixture's leg is a rounded + # sine, which makes its own steps lumpy (1.50 1.20 0.32 0.32 1.20 1.50 ... x the mean), + # so the closest any cut can put the wrap to an average step here is ~1.2. + assert 0.9 < cycle["ratio"] < 1.6 def test_detect_cycle_refuses_empty_window() -> None: @@ -455,7 +461,7 @@ def fake_video(image, prompt, out, report, *, duration, resolution, log): monkeypatch.setattr(batch_mod.time, "sleep", lambda s: None) # no real backoff waits monkeypatch.setattr(frames_mod, "run_frames", lambda clip, out_dir, **kw: {"fps": 24.0, "frames": 10, "alpha_zero_pct_min": 60.0, "alpha_zero_pct_max": 70.0, "keyed_dir": str(out_dir / "keyed")}) monkeypatch.setattr(batch_mod.frames_mod, "run_frames", frames_mod.run_frames) - monkeypatch.setattr(batch_mod.loop_mod, "run_loop", lambda frames_dir, out_dir, **kw: {"cycle": {"length": 12, "period_global": 12, "ratio": 0.3}, "resampled_seam_ratio": 0.5, "n_out": 12, "gif": {"file": "x.gif"}, "webp": {"file": "x.webp"}, "strip": {"path": "x.png"}}) + monkeypatch.setattr(batch_mod.loop_mod, "run_loop", lambda frames_dir, out_dir, **kw: {"cycle": {"length": 12, "period_global": 12, "ratio": 0.3, "review_recommended": kw["state"] == "run"}, "resampled_seam_ratio": 0.5, "n_out": 12, "gif": {"file": "x.gif"}, "webp": {"file": "x.webp"}, "strip": {"path": "x.png"}}) payload = batch_mod.run_set(bases={"side": base}, states=["walk", "run", "jump"], root=tmp_path / "set", character=None, duration=6, resolution="720p", key="green", concurrency=3, force=False, gap=0.0, video_runner=fake_video) @@ -463,8 +469,11 @@ def fake_video(image, prompt, out, report, *, duration, resolution, log): assert by["side-walk"]["ok"] and by["side-walk"]["clip"]["attempts"] == [0] assert by["side-run"]["ok"] and by["side-run"]["clip"]["attempts"] == [1, 0] # one 429 retry assert not by["side-jump"]["ok"] and "clip generation failed" in by["side-jump"]["error"] + assert by["side-run"]["loop"]["review_recommended"] is True + assert by["side-walk"]["loop"]["review_recommended"] is False assert payload["failed"] == ["side-jump"] table = (tmp_path / "set" / "table.md").read_text() assert "| side | jump | - | - | - | - | - | FAIL" in table and "| side | walk | periodic | 12 | 12 | 0.50 | 12 | OK |" in table + assert "| side | run | periodic | 12 | 12 | 0.50 | 12 | OK (review gait) |" in table assert (tmp_path / "set" / "set.report.json").is_file() assert (tmp_path / "set" / "side-walk" / "canvas.png").is_file() # canvas ran for real