Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AUTHORS
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,4 @@ Bassam Adnan <mailbassam@gmail.com>
Saleh Enab <salehenab850@gmail.com>
Tejas Tyagi <tejastyagi.tt@gmail.com>
abishekcs <abishekdascs@gmail.com>
Adnan Khan <adnan.kriaz@gmail.com>
22 changes: 14 additions & 8 deletions doc/manual/en/46-server.md
Original file line number Diff line number Diff line change
Expand Up @@ -4328,7 +4328,10 @@ arrived in: a JPEG stays a JPEG, a PNG a PNG, a GIF a GIF (one frame, 256
colours), a WebP a lossless WebP, an SVG an SVG — drawn first, and the
picture comes back as an SVG document carrying it as PNG, there being no
pixels-to-vector. An animated GIF or WebP is read as its first frame. The attachment itself shows as a thumbnail under
your message.
your message. One picture can be attached; attaching several is
currently rejected (the limits are described under
*POST /v1/images/generations* in the _HTTP endpoints_ chapter, and
support for models with reference conditioning follows).

Attaching a picture to a _language_ model's chat sends it to the model when
the model reads pictures (see **Pictures in a chat**); otherwise the model is
Expand All @@ -4341,13 +4344,16 @@ contents.
`size`, `output_format` (`png`, `jpeg`, `gif` or `webp`), `response_format: "b64_json"`
— plus the knobs above per request (`steps`, `cfg_scale`,
`negative_prompt`, `seed`) and, in place of OpenAI's multipart
`/v1/images/edits`, an `image` (a `data:` URL or bare base64 of a PNG,
JPEG, GIF, WebP or SVG) with a `strength` to start from. `stream: true` reports a
step at a time. `POST /v1/chat/completions` on an image server answers the
last user message with the picture as a markdown image over a `data:` URL
— which any chat client that renders markdown shows inline — and takes a
picture from an `image_url` content part. Both are documented field by
field in the _HTTP endpoints_ chapter.
`/v1/images/edits`, an `image` or an `images` list (a `data:` URL or
bare base64 of a PNG, JPEG, GIF, WebP or SVG; see the _HTTP endpoints_
chapter for limits; more than one is currently rejected) with a
`strength` to start from. `stream: true` reports a step at a time.
`POST /v1/chat/completions` on an image server answers the last user
message with the picture as a markdown image over a `data:` URL —
which any chat client that renders markdown shows inline — and collects
all `image_url` content parts in order (more than one is currently
rejected). Both are documented field by field in the _HTTP endpoints_
chapter.

## Video generation

Expand Down
18 changes: 12 additions & 6 deletions doc/manual/en/48-image.md
Original file line number Diff line number Diff line change
Expand Up @@ -396,7 +396,10 @@ the prompt edits it — see **Editing a picture** above. With Qwen-Image
attachment rather than from noise, running *Strength* of the schedule — a
rough sketch redrawn as a painting, or a photograph in another style.
Either way the picture keeps the attachment's proportions and comes back
in its format.
in its format. One picture can be attached; attaching several is
currently rejected (the limits are described under
*POST /v1/images/generations* in the *HTTP endpoints* chapter, and
support for models with reference conditioning follows).

A long picture can be stopped with the console's Stop button, or by
closing the tab: the work ends within seconds, at the next transformer
Expand All @@ -416,12 +419,15 @@ Everything the console does, a client can:

- `POST /v1/images/generations` is OpenAI's Images API with the local
model's knobs added (`size`, `steps`, `cfg_scale`, `negative_prompt`,
`seed`, `output_format`, and an `image` with `strength` to start
from). `stream: true` sends a progress event per step, the first one
— step 0 — carrying the estimate before any work.
`seed`, `output_format`, and an `image` or `images` list with
`strength` to start from; see the *HTTP endpoints* chapter for limits;
more than one is currently rejected). `stream: true` sends a progress
event per step, the first one — step 0 — carrying the estimate before
any work.
- `POST /v1/chat/completions` on the same server treats the last user
message as the prompt, and its last `image_url` part as the picture to
start from; the answer is the picture as a markdown image.
message as the prompt, and collects all `image_url` parts in order as
pictures to start from (the same limits apply; more than one is
currently rejected); the answer is the picture as a markdown image.
- `GET /props` reports the companions, the adapter, the defaults, the
measured rate and what a picture costs at it; `POST /props` sets the
defaults — it is what the Image pane's Save calls.
Expand Down
29 changes: 20 additions & 9 deletions doc/manual/en/80-http.md
Original file line number Diff line number Diff line change
Expand Up @@ -788,9 +788,18 @@ curl -s localhost:8100/v1/images/generations -H 'Content-Type: application/json'
| `steps` | `image_steps` | denoising steps |
| `cfg_scale` | `image_cfg_scale` | classifier-free guidance; `1` runs the prompt alone, half the work |
| `image` | — | a picture to start from — a `data:image/…;base64,…` URL or bare base64 of a PNG, JPEG, GIF, WebP or SVG. Resized to `size`. On a `qwen_image_2_1` server with a vision projector (`/props`' `image.vision`), the picture is **edited** by the prompt instead: read by the text encoder and kept wherever the prompt does not ask for a change |
| `images` | — | an array of pictures, in order; use it instead of `image`; supplying both is an error |
| `strength` | `image_strength` | with `image`: how much of the schedule to run, `0` (unchanged) to `1` (the picture's content is ignored). Not read for an edit |
| `stream` | `false` | server-sent events instead of one body |

A request takes at most 4 pictures (as `image` or in `images`). When more
than one is provided, their combined size is capped at 100 MB and 32
megapixels, and errors name the image (for example `image at index 2`).
With `size` omitted or `auto`, the last image sets the proportions; an
explicit size overrides. More than one image is currently rejected on
every model; support for models with reference conditioning follows, and
a single image is unaffected.

The reply is OpenAI's: `{"created", "model", "data": [{"b64_json", …}]}`,
with each picture also carrying `output_format`, `size`, `seed`, `steps`,
`generation_ms` and `timings` — `{"encode_ms", "steps_ms", "decode_ms"}`,
Expand Down Expand Up @@ -820,16 +829,18 @@ as `token_passes_per_second` (at the defaults' size) and
and the rate model itself — see **`GET /props`** below.

`POST /v1/chat/completions` on the same server draws from the last user
message: its text is the prompt, and the last `image_url` part of an
message: its text is the prompt, and all `image_url` parts of an
array-shaped `content` (`{"type": "image_url", "image_url": {"url":
"data:image/png;base64,…"}}`) is the picture to start from. Every knob is
the server's `image_*` default except `seed`. The assistant's `content` is
the picture as a markdown image over a `data:` URL, which a markdown-rendering
chat client shows inline; `message.images` carries the same picture as
`b64_json` with `mime`, `size`, `seed`, `steps` and `generation_ms` for a
client that wants the bytes. Streamed, the chunks carry an empty `delta` and
an `image_progress` object (`step`, `steps`, `eta_seconds`) until the final
chunk delivers the content with `finish_reason: "stop"`.
"data:image/png;base64,…"}}`) are collected in order as pictures to
start from (the same limits apply; more than one is currently rejected).
Every knob is the server's `image_*` default except `seed`. The
assistant's `content` is the picture as a markdown image over a `data:`
URL, which a markdown-rendering chat client shows inline;
`message.images` carries the same picture as `b64_json` with `mime`,
`size`, `seed`, `steps` and `generation_ms` for a client that wants the
bytes. Streamed, the chunks carry an empty `delta` and an `image_progress`
object (`step`, `steps`, `eta_seconds`) until the final chunk delivers
the content with `finish_reason: "stop"`.

#### `POST /v1/videos`

Expand Down
10 changes: 10 additions & 0 deletions src/bin/orangu-server/engine/image/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -681,6 +681,9 @@ pub struct ImageRequest {
pub format: ImageFormat,
/// A picture to start from, for image-to-image.
pub init: Option<InitImage>,
/// Pictures to start from, for image-to-image and multi-reference editing.
#[allow(dead_code)]
pub inits: Vec<InitImage>,
}

#[derive(Debug, Clone)]
Expand Down Expand Up @@ -1483,6 +1486,12 @@ impl Pipeline {
self.vision.is_some()
}

/// Whether this pipeline supports multiple reference images.
/// In PR 1 this returns false on all models; PR 2 will wire this to `self.edits()`.
pub fn supports_multi_reference(&self) -> bool {
false
}

/// An attached picture prepared for editing a `(width, height)`
/// picture: read by the vision tower, and encoded (the posterior's
/// mean, as diffusers takes it) to latents.
Expand Down Expand Up @@ -2518,6 +2527,7 @@ mod tests {
seed: Some(1),
format: ImageFormat::Png,
init: None,
inits: Vec::new(),
},
&mut |p| {
steps_seen = p.step;
Expand Down
Loading
Loading