diff --git a/PROVIDERS.md b/PROVIDERS.md index 38cd8b4..aa64e12 100644 --- a/PROVIDERS.md +++ b/PROVIDERS.md @@ -27,7 +27,7 @@ them. | PixelLab | Production; paid generation and account workflows live-tested | `PIXELLAB_API_KEY` | generations | | Retro Diffusion | Experimental; authenticated paid still generation, download, provenance, and recovery live-tested; advanced workflows pending | `RD_API_KEY` | USD | | ComfyUI | Experimental; local generation, review, recovery, and image-to-image revision smoke live-tested on Apple MPS; inpaint/outpaint remain mock-tested | none; optional `COMFYUI_BASE_URL` | free | -| Scenario | Experimental; BFL Flux 2 Dev authentication, CU preflight, paid single/two-output generation, review, download, and durable recovery live-tested | `SCENARIO_SDK_API_KEY` and `SCENARIO_SDK_API_SECRET` | compute-units | +| Scenario | Experimental; BFL Flux 2 Dev authentication, CU preflight, paid single/two-output generation, review, download, and durable recovery live-tested; reference-image edits, tool models, and multi-model chains run once each | `SCENARIO_SDK_API_KEY` and `SCENARIO_SDK_API_SECRET` | compute-units | | FakeProvider | Test-only deterministic lifecycle | none | free | Live tests now cover single-candidate RD Fast and RD Plus stills from cost quote @@ -267,9 +267,11 @@ to ComfyUI. Provider-keyed budgets, independent orchestration, lock-authoritativ recovery, and explicit account-provider selection ship with it. The Scenario still-image adapter from -[GitHub issue #52](https://github.com/gfargo/pixelkiln/issues/52) now has mocked -edge coverage and a paid BFL Flux 2 Dev lifecycle smoke. It remains -experimental until more model schemas and representative art briefs pass. +[GitHub issue #52](https://github.com/gfargo/pixelkiln/issues/52) has mocked edge +coverage, a paid BFL Flux 2 Dev lifecycle smoke, and a broader evaluation (ten +more models, six tool models, reference-image edits, multi-model chains, and a real +game's art) recorded in [Set up Scenario](./docs/SCENARIO.md#where-the-evidence-is). +It remains experimental until more model schemas and representative art briefs pass. | Candidate | What it adds | Fit with PixelKiln | Main cost or risk | Priority | |---|---|---|---|---:| diff --git a/README.md b/README.md index 3f514e3..daf46c7 100644 --- a/README.md +++ b/README.md @@ -366,7 +366,7 @@ verification. See [Library API](./docs/LIBRARY.md). | [Set up PixelLab](./docs/PIXELLAB.md) | Production-provider credentials, manifest, generators, and account workflows. | | [Set up Retro Diffusion](./docs/RETRO_DIFFUSION.md) | Experimental-provider credentials, styles, formats, cost checks, and limits. | | [Set up ComfyUI](./docs/COMFYUI.md) | Self-hosted stills, revisions, ordered frame sets, per-asset inputs, and quality limits. | -| [Set up Scenario](./docs/SCENARIO.md) | Experimental hosted models, two-part credentials, CU preflight, review, and durable downloads. | +| [Set up Scenario](./docs/SCENARIO.md) | Experimental hosted models, two-part credentials, CU preflight, reference-image edits, multi-model chains, measured findings, and durable downloads. | | [Versioned recipes](./docs/RECIPES.md) | Pinned workflow packs, model hashes, manifest templates, and quality contracts. | | [Controlled revisions](./docs/REVISIONS.md) | Image-to-image, inpaint, palette cleanup, animation, and interpolation from a parent; masks, fail-closed readiness, provenance, and ComfyUI bindings. | | [CLI reference](./docs/CLI.md) | Every command, flag, JSON mode, and exit contract. | diff --git a/docs/AGENTS.md b/docs/AGENTS.md index 4a79603..cd538b8 100644 --- a/docs/AGENTS.md +++ b/docs/AGENTS.md @@ -72,7 +72,7 @@ replacement: | PixelLab adapter | The current production and live-tested generation backend: stills, tiles, UI panels and elements, `objectPro` objects, controlled revisions, and the character family (four engines, states, loops, mirrors, portraits, outfits, adoption). | | Retro Diffusion adapter | Experimental backend; authenticated paid single-still lifecycle plus mocked advanced-workflow tests. | | ComfyUI adapter | Experimental self-hosted still, controlled-revision, and atomic frame-set backend; PixelKiln automates lineage and mechanical quality checks while a person retains visual approval. | -| Scenario adapter | Experimental hosted still-image backend; BFL Flux 2 Dev quote, paid single/two-output generation, human review, and durable recovery live-tested. | +| Scenario adapter | Experimental hosted still-image backend with reference-image edits and cross-style chains; BFL Flux 2 Dev quote, paid generation, human review, and durable recovery live-tested, plus ten more models, six tool models, and a real game's art run once each. | PixelKiln's core is provider-neutral, but PixelLab remains the only production adapter. Retro Diffusion generation support is experimental. Paid RD Fast and diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index d45c666..276b0af 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -288,7 +288,7 @@ coverage. Live Apple MPS runs cover a small Stable Diffusion 1.5 plumbing smoke and the SDXL Base plus Pixel Art XL composition benchmark, including four-candidate review, cache recovery, and native-grid refinement. `ScenarioProvider` is an experimental hosted still-image -adapter. It keeps offline planning conservative, records the provider's free +adapter that also uploads references for `styleImages` and `image-to-image` revisions. It keeps offline planning conservative, records the provider's free submit-time Compute Unit quote, and stores durable `scenario://` job and asset references that refresh signed URLs. Authentication and CU preflight have passed live with BFL Flux 2 Dev, along with paid single/two-output jobs, human diff --git a/docs/CLI.md b/docs/CLI.md index d0cfd20..fe94503 100644 --- a/docs/CLI.md +++ b/docs/CLI.md @@ -17,7 +17,7 @@ work accordingly. The experimental `retrodiffusion` adapter supports still-image `map`/`pixflux`, `tiles` sheets, and `animation` GIF/spritesheet work. The experimental `comfyui` adapter runs committed API-format `map` and ordered still-frame workflows on a self-hosted server. The experimental `scenario` adapter runs -hosted still models with Compute Unit preflight and durable asset recovery. +hosted still models, with reference-image edits, Compute Unit preflight, and durable asset recovery. Which command: diff --git a/docs/GETTING_STARTED.md b/docs/GETTING_STARTED.md index 6a4968b..a6985ef 100644 --- a/docs/GETTING_STARTED.md +++ b/docs/GETTING_STARTED.md @@ -73,7 +73,7 @@ remains the production adapter. See provider options and current limits, or [provider comparison](../PROVIDERS.md) for selection guidance. -For experimental Scenario still generation, use both credentials: +For experimental Scenario still generation and reference-image edits, use both credentials: ```dotenv SCENARIO_SDK_API_KEY=... diff --git a/docs/SCENARIO.md b/docs/SCENARIO.md index c6783ea..2b472ec 100644 --- a/docs/SCENARIO.md +++ b/docs/SCENARIO.md @@ -1,16 +1,29 @@ # Set up Scenario [Scenario](https://www.scenario.com/) is an experimental hosted provider for -project-specific models and third-party image models. PixelKiln currently -supports still PNG generation through Scenario's universal model endpoint, -free Compute Unit preflight, asynchronous jobs, multi-output review, and -refreshable asset downloads. - -Every code path in the adapter has mocked coverage. BFL Flux 2 Dev has also passed -live authentication, cost preflight, paid single- and two-output generation, -human review, PNG download, and provider-backed recovery. Start every untested -model with one disposable asset and a small ceiling. This integration remains -experimental because Scenario model schemas vary. +project-specific models and third-party image models. PixelKiln supports still +PNG generation through Scenario's universal model endpoint, free Compute Unit +preflight, asynchronous jobs, multi-output review, and refreshable asset +downloads. It also uploads your images as references, so a style's +`styleImages` and `image-to-image` revisions work, and a manifest can chain +several Scenario models across styles with +[`revision.fromStyle`](./REVISIONS.md#across-styles). The adapter handles +still images only; audio models are reachable through the same account but not +through PixelKiln. + +Every code path in the adapter has mocked coverage. BFL Flux 2 Dev passed live +authentication, cost preflight, paid single- and two-output generation, human +review, PNG download, and provider-backed recovery. A later evaluation ran ten +more text-to-image models, six tool models, reference-image edits, multi-model +chains, and a real game's art; its findings are on this page and its benchmark +projects are listed in [Where the evidence is](#where-the-evidence-is). Start +every untested model with one disposable asset and a small ceiling. This +integration remains experimental because Scenario model schemas vary. + +The short version of what that evaluation found: price does not predict fit; +large or illustrative art (landmarks, covers, flat backdrops) comes out well +through render, Pixelate, cutout, and `refine`; small native-size sprites, tiles, +and characters do not; and a reference image keeps an existing design. ## Requirements @@ -371,6 +384,57 @@ and post steps beside the file. Music models (ACE-Step, MusicGen, ElevenLabs Music, Lyria 3.5) quoted 10 to 30 CU; Sonilo and Lyria 3 Pro need the `cu-pro-q3-25` plan. Music was not generated. +### Calling the API directly: the dry run and the legacy endpoint + +Two facts for anyone calling Scenario outside PixelKiln, because both cost compute +units to find: + +- **The free dry run is only the query parameter** `?dryRun=true`. A `dryRun: true` + in the JSON body is ignored, and the request submits and bills a real job. A probe of + the legacy `/generate/txt2img` endpoint made that mistake and billed 10 CU for a + FLUX.1 LoRA image, and jobs there cannot be cancelled ("Cannot cancel this type of + job"). The adapter always sends the query parameter. +- **LoRA models do not use `/generate/custom/{modelId}`.** A public pixel-art LoRA + (Flux Retro Aesthetics) answered "Custom models only are supported for this + endpoint" there. The older `/generate/txt2img` endpoint accepted it at 10 CU an + image, returned a JPEG (which the adapter refuses), and is absent from the current + API reference. The adapter does not call it, and the same-quality Klein render costs + 1 CU. + +### Budget and billing notes + +- **The adapter cannot read your balance.** `gen` enforces the budget you pass + (`scenario does not expose an account balance; enforcing its run budget`) and + the per-style `maxComputeUnits`, but the remaining balance is only on Scenario's + dashboard. Whether the allowance renews depends on the plan and billing + setup; a dashboard showing "No monthly reset scheduled" means it may not. +- **Check the dashboard against your own tally.** In our evaluation a hand total + of quoted and billed amounts came to about 715 CU while the dashboard's balance + had fallen by 753 CU, about 5% more. The cause was not identified, so budget with a + margin. +- **Quotes were exact where the adapter made them.** Every quote the adapter took + matched the billed amount in the lockfile, including failed assets that the + adapter refused after billing (a JPEG or WebP result is billed even though no + file lands). +- **Flat prices.** GPT Image 2 `quality: medium` was 11 CU at any canvas size (`high` + 44), ElevenLabs sound effects 30 CU at any length, and tool models 2 to 10 CU. + +### Where the evidence is + +Each is a committed project with its manifest, lockfile, outputs, and README: + +| Project | What it ran | CU | +|---|---|---:| +| `benchmarks/provider-scenario-smoke` | The first live run: FLUX.2 Dev, one and two outputs, recovery | 48 | +| `benchmarks/provider-scenario-bakeoff` | Ten models on one brief | 33 | +| `benchmarks/provider-scenario-edit` | One reference-image edit | 1 | +| `benchmarks/provider-scenario-tools` | Birefnet, Pixel Snapper, Pixelate, three upscalers, and the refine gate | 34 | +| `benchmarks/provider-scenario-pipeline` | Generate, pixelate, cut out, refine a character, with a sweep of Pixelate grid sizes | 41 | +| `benchmarks/provider-scenario-chain` | The same chain declared with `revision.fromStyle` and run by one `gen` | 8 | +| `benchmarks/provider-scenario-game-pilot` | A real game's landmarks, covers, backdrops, structures, and sound effects (settings and measurements only; the game's art is not included) | about 630 | + +Scratch projects for the game pilot are not committed, so its README is the record. + ## Plan before spending Scenario prices depend on model, size, steps, and output count, so PixelKiln @@ -523,10 +587,13 @@ quote, final billing, selected asset ID, output index, and output hash without retaining credentials or signed query strings. Once cached, `restore` prefers the validated local content cache. -Scenario account listing, adoption, salvage, tagging, deletion, balance, image -uploads, training, background removal, and upscaling are not part of this first -adapter. PixelKiln reports unavailable account commands instead of pretending -they succeeded. +Scenario account listing, adoption, salvage, tagging, deletion, balance, and +training are not part of the adapter. Image uploads exist only to send +references (see [Reference images and edits](#reference-images-and-edits)), and +background removal, pixel snapping, and upscaling run as ordinary models through +`image-to-image` revisions ([Tool models](#tool-models-as-image-to-image)). +PixelKiln reports unavailable account commands instead of pretending they +succeeded. ## Mixed-provider projects diff --git a/skills/pixelkiln/references/scenario.md b/skills/pixelkiln/references/scenario.md index ef14b47..7306da6 100644 --- a/skills/pixelkiln/references/scenario.md +++ b/skills/pixelkiln/references/scenario.md @@ -27,7 +27,7 @@ schemas remain unverified. Require a one-asset smoke before widening a batch. `referenceArray` for a model that names it differently. Other revision modes and `strength` are refused. -Scenario model parameters vary. The MVP supplies prompt, 16–4096px dimensions +Scenario model parameters vary. The adapter supplies prompt, 16–4096px dimensions (each model has tighter bounds, reported by the free dry run), one to four outputs, and optional seed. Put only documented model inputs in `parameters`; PixelKiln rejects attempts to override its prompt, dimensions, seed, output count, project routing, or CU ceiling. @@ -45,5 +45,14 @@ See `docs/SCENARIO.md` for settings and measurements. PixelKiln validates the job, bytes, hashes, and recovery path. It does not prove the output is good pixel art. Inspect one result at 1× before approving a batch. +When calling Scenario outside PixelKiln (audio, say), the free dry run is only the +query parameter `?dryRun=true`: a `dryRun: true` in the JSON body is ignored and +submits and bills a real job. LoRA models need the legacy `/generate/txt2img` +endpoint (10 CU, JPEG output, not callable by the adapter). The adapter cannot read +the CU balance; check the dashboard and budget with a margin (a hand tally ran about +5% under the dashboard). A reference image keeps an existing design (silhouette +overlap 0.93 to 1.00 against 0.39 to 0.66 from text alone), though some models shift +its colours; use it before a text-only prompt when the look must be kept. + See `docs/SCENARIO.md` for setup, manifest examples, unsupported account operations, and the live validation checklist. diff --git a/website/app/globals.css b/website/app/globals.css index a5c0e3c..e5f71ac 100644 --- a/website/app/globals.css +++ b/website/app/globals.css @@ -768,6 +768,23 @@ a { color: inherit; text-decoration: none; } line-height: 1.4; } .scenario-strip figcaption b { display: block; color: var(--ember-light); font-weight: 600; } +.scenario-findings { margin-top: 18px; } +.scenario-findings ul { + display: grid; + grid-template-columns: repeat(2, minmax(0, 1fr)); + gap: 0 34px; + margin: 0; + padding: 0; + list-style: none; +} +.scenario-findings li { + padding: 15px 0; + border-top: 1px solid var(--line); + color: #aaa293; + font-size: 13px; + line-height: 1.65; +} +.scenario-findings li b { color: #e6dccb; font-weight: 600; } .scenario-bottom { display: grid; grid-template-columns: minmax(0, 1.1fr) minmax(0, 1fr); @@ -1083,7 +1100,7 @@ a { color: inherit; text-decoration: none; } .review-grid { grid-template-columns: 1fr; } .review-copy { max-width: 720px; } .provider-showcase-grid { grid-template-columns: 1fr; } - .scenario-grid, .scenario-bottom { grid-template-columns: minmax(0, 1fr); } + .scenario-grid, .scenario-bottom, .scenario-findings ul { grid-template-columns: minmax(0, 1fr); } .lock-visual { width: min(680px, calc(100% - 35px)); } .docs-layout { grid-template-columns: 170px minmax(0, 1fr); } .docs-toc { display: none; } diff --git a/website/app/page.tsx b/website/app/page.tsx index a5fbdba..5a80d51 100644 --- a/website/app/page.tsx +++ b/website/app/page.tsx @@ -260,9 +260,10 @@ export default function Home() { and grid refinement. Atomic frame sets have automated coverage, and accepted frame prompts are saved before the next one is queued. They still need a live pose recipe and benchmark. The tested SDXL - graph needs manual cleanup and art review. Scenario's BFL profile - has passed paid single- and two-output generation, human review, - and durable recovery. + graph needs manual cleanup and art review. Scenario has passed paid + single- and two-output generation, human review, and durable + recovery, and has run ten more models, six tool models, + reference-image edits, and multi-model chains once each.

@@ -429,7 +430,8 @@ export default function Home() { pixel grid, because a large raster can still contain fake pixels. Its samples are diagnostics, not finished asset recommendations. Scenario shows its first paid integration smoke, not a comparable - environment benchmark. A manifest style.quality profile + environment benchmark; its measured findings are in their own + section below. A manifest style.quality profile can apply the same offline palette and approval gate after any supported single-image provider or an atomic ComfyUI frame set. ComfyUI and PixelLab both back controlled revisions, and @@ -635,8 +637,8 @@ export default function Home() {

Measured cost
16 CU for one output; 32 CU for two
-
Best fit
Hosted models and project-specific LoRA experiments
-
Readiness
One BFL profile live-tested; other schemas unverified
+
Best fit
Large or illustrative art, reference-image edits, and chains of hosted models
+
Readiness
Eleven image models, six tool models, and reference edits run once each; other schemas unverified
@@ -742,6 +744,19 @@ export default function Home() {
+
+

What a real game taught us

+
    +
  • Large art works. A 240 px landmark chain returned the existing sprite's exact size at high confidence, with far more detail, for 8 CU on the cheap model or 18 CU on the strongest.
    • +
  • A reference keeps the design. Sending the current sprite as a reference raised silhouette overlap from 0.39 to 0.66 to 0.93 to 1.00, at the same pixel scale. Some models shift colours.
    • +
  • Small sprites and tiles do not. 32 px props came back as different objects with holes after the cutout; terrain tiles and characters need tools built for them.
    • +
  • Backdrops need the prompt to say flat. An isometric prompt lay on a diagonal plane; asking for a front-facing horizontal strip gave the exact grid. A mist fade needs a better cutout route.
    • +
  • The pixelate grid is not always literal. On a 960 px render, asking for 240 made 2 px blocks and asking for 120 made the 240 cells wanted. Check one image first.
    • +
  • Sound effects need one event each. A two-part impact worked only as two sounds layered, and only on the pricier model; the cheap one returned ambience.
    • +
  • Quotes were exact, the balance is not visible. Every quote matched its bill, but the adapter cannot read the account balance, and a hand tally ran about 5% under it.
    • +
+
+ 
{`"assets": {