EPIC: Local ComfyUI character likeness + style transfer without LoRA (FLUX-first) — testing regimen

[taskmasterpeace]

EPIC: Local ComfyUI image generation — character likeness + style transfer WITHOUT LoRA (FLUX-first)

The goal (why this matters)

Get reliable character likeness AND art-style transfer in generated images on cupcake's
local ComfyUI — without training a LoRA and without paying Nano Banana / GPT-image.
Optionally stack existing FLUX LoRAs when we have them. If this works, Directors Palette /
Directors Desktop can generate on-model character images (and character sheets) locally and free.

Runs on cupcake (GB10, sm_121, 131GB) — the same box now doing the video benchmarks. This is
the image sibling of the video engine work (Directors-Desktop#86/#87, directors-palette-v2#219).

The core architectural shift (from the research)

These are identity-adapter tools, not pixel editors like Qwen-Image-Edit. They extract a
face embedding from reference photo(s) and inject it as conditioning into a fresh generation —
so identity and style are controlled on independent branches. That separation is why good
ComfyUI results beat single-model edits: the face branch never carries the art style, the style
branch never has to identify the person.

Target pipeline = 3 independent branches + prompt:

Branch	Tool	Controls
Identity	1–3 face refs → PuLID-FLUX / InfiniteYou / IPAdapter-FaceID	facial likeness
Style	style board/pack → IPAdapter Plus/Style	palette, medium, linework, texture, lighting
Pose/comp	pose img/sketch → ControlNet OpenPose/Depth/Canny	what the character is doing + framing
Scene	text prompt	action, setting, props, mood — NOT facial traits (identity branch supplies those)

Methods to test (all no-LoRA)

Method	Base	Note
PuLID-FLUX	FLUX.1-dev	best identity/text balance in benchmarks; engineered against "frozen face". Heavy deps (EVA-CLIP, InsightFace, AntelopeV2, facexlib)
InfiniteYou (InfU)	FLUX	paper beat PuLID-FLUX 72.8% vs 27.2% overall preference — test at EQUAL priority to PuLID
IPAdapter FaceID Plus V2	SDXL	easiest install, most community docs; can freeze pose if weight too high. Run InsightFace on CPU (VRAM saver)
InstantID	SDXL	fast, tuning-free; text-editability tradeoff vs PuLID

FLUX-first per Robert (we already own FLUX LoRAs). SDXL methods (IPAdapter/InstantID) are the
fast-to-stand-up baselines.

Style transfer sub-question (Robert's "style-die-sheet" idea)

Can a single 6-panel style board (horse / underwater / blonde-on-beach / apple-on-desk / living
room / city skyline) teach the style? Research says: it CAN work as a broad anchor, but a curated
style pack (3 strong separate images: a portrait, an environment, an object/creature) is more
controllable than one collage (the model may copy the collage layout or over-weight the biggest
tile). Test both. Clean 2×3 board, no labels/borders/text.

Test assets (Robert-provided)

• Identity/likeness refs: the Palette character sheets for @Hitman Holla and @AyeVerb
  (two different sheet styles) — do our sheets read as usable identity refs?
• Style ref: the comic-book still (woman in red dress + cowboy w/ shotgun, mall) — target art style.
• Test scene + baseline-to-beat: "close-up of a man sitting in a cafe looking out the window."
  Robert's Qwen-Image-Edit attempt is the BASELINE WE'RE BEATING (waxy skin, weak likeness).
  Also a film still (older man, office, paper bag) as an identity target.

The test matrix (same rigor as the video battery — same face, same style, same scene, same seed)

#	Vary	Score
1	Face-lock baseline (neutral prompt, plain bg)	identity match only
2	Style injection (add style branch)	does style apply, or face "photo-pasted" onto stylized scene?
3	Adapter weight sweep (0.4/0.6/0.8/1.0)	find where pose "freezes" to the ref photo
4	Pose/action change (4 verbs)	does identity hold when pose changes?
5	Cross-model (PuLID / InfU / IPAdapter-FaceID / InstantID, same 3 inputs)	side-by-side ranking on OUR assets
6	Text-editability (unusual lighting/props)	does the adapter fight the prompt?
+	Style test 4-way: text-only / one hero image / 6-panel board / 3-image batch	style match across person+horse+city+water; did it copy a panel's composition?

Scoring (1–5 each, log adapter weight + base + steps + seed every run): identity fidelity ·
style fidelity (art style, not a photo w/ filter) · prompt/scene compliance · artifact/overfitting
(waxy skin, frozen expression, pose bleed).

Expected outcome (from published comparisons)

InfiniteYou + PuLID-FLUX should beat IPAdapter-FaceID on likeness-under-stylization (both built for
the face-lock + style-freedom tradeoff). Start IPAdapter-FaceID for fastest setup; prioritize
PuLID-FLUX + InfiniteYou for best quality.

Deliverable

Same as the video work: a filled scorecard/playground + a benchmarks/image/ record on cupcake +
final recommendations fed to Palette (#219) and DD so the winning workflow becomes a provider.
Success = a local, free, on-model character image that beats the paid Nano/GPT-image baseline.

Sub-issues

• PuLID-FLUX install + face-lock baseline on cupcake
• InfiniteYou install + baseline (equal priority to PuLID)
• IPAdapter FaceID Plus V2 (SDXL) — fast baseline
• InstantID (SDXL)
• Style-transfer branch — board vs pack vs hero-image (IPAdapter Plus/Style)
• Combined 3-branch pipeline (identity + style + ControlNet pose) + the scoring regimen
• Optional: FLUX LoRA stacking on the winning identity method

[taskmasterpeace]

Sub-issues created:

• [likeness] PuLID-FLUX install + face-lock baseline on cupcake #89 PuLID-FLUX (FLUX-first primary)
• [likeness] InfiniteYou (FLUX) install + baseline — equal priority to PuLID #90 InfiniteYou (equal priority — beat PuLID 72.8/27.2 in paper)
• [likeness] IPAdapter FaceID Plus V2 (SDXL) — fastest baseline #91 IPAdapter FaceID Plus V2 (SDXL, fastest baseline)
• [likeness] InstantID (SDXL) #92 InstantID (SDXL)
• [style] Style-transfer branch — board vs pack vs hero image (IPAdapter Plus/Style) #93 Style-transfer branch (board vs pack vs hero)
• [likeness] Combined 3-branch pipeline (identity + style + ControlNet pose) + scoring regimen #94 Combined 3-branch pipeline + scoring regimen

Runs on cupcake (same box as the video benchmarks). Suggested order: #91 (fast to prove the concept) → #89 + #90 (the FLUX quality contenders) → #93 (style) → #94 (combine + score) → #92 fills the cross-model matrix. Winning workflow → Palette #219 as a local provider.