Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 32 additions & 10 deletions docs/dcc-submitters.md
Original file line number Diff line number Diff line change
Expand Up @@ -230,17 +230,21 @@ enforcement point, and the submitter must not break against an older server.

## Reference presets

sqi ships six DCC reference products under `presets/sqi/`, published through
sqi ships nine DCC reference products under `presets/sqi/`, published through
the [preset library](preset-library.md) so `sqi-submitter` has something real
to submit to out of the box (the directory holds eleven products in total —
to submit to out of the box (the directory holds fourteen products in total —
the other five are `Transcoding`-category ffmpeg presets; see [Transcoding
reference presets](preset-library.md#transcoding-reference-presets)). Install
them from **Admin → Preset Library** in the web UI, or by hand via `POST
/api/v1/products` (or **Admin → Products** in the web UI) if you're not using
a preset index. All use the product `category`
`Rendering` and declare `TASK_CHUNKING` for frame distribution; the Maya and
Blender presets additionally declare `SQI_CHUNK_BOUNDS` to expose each chunk's
start/end frame (see [OpenJD extensions](openjd-extensions.md)).
`Rendering` and declare `TASK_CHUNKING` for frame distribution; the Maya,
Blender and Mistika presets additionally declare `SQI_CHUNK_BOUNDS` to expose
each chunk's start/end frame (see [OpenJD extensions](openjd-extensions.md)).

Six of the nine have a matching `sqi-submitter` host (Maya, Houdini, Nuke,
Blender). The three Mistika presets do not — Mistika has no in-DCC submitter,
so they are submitted from the web UI or the API like any other product.

| Product | Command | Chunking | Parameters | Worker tag |
|---|---|---|---|---|
Expand All @@ -250,6 +254,9 @@ start/end frame (see [OpenJD extensions](openjd-extensions.md)).
| `nuke-write-render` | `nuke -x -X <writeNode> -F <range> <script>` | chunks of 10 | `SceneFile`, `Frames`, `WriteNode` | `attr.worker.tag.nuke = "true"` |
| `nuke-script-render` | `nuke -x -F <range> <script>` (no `-X`; all enabled Write nodes) | chunks of 10 | `SceneFile`, `Frames` | `attr.worker.tag.nuke = "true"` |
| `blender-batch-render` | `blender -b <file> -o <output> -s/-e <frame range> -a` | one frame per task by default (raise the chunk size to batch) | `SceneFile`, `Frames`, `OutputPath` (optional; blank = scene setting) | `attr.worker.tag.blender = "true"` |
| `mistika-boutique-render` | `mistika -r <scene> -s/-e <frame range>` | chunks of 10 | `SceneFile` (`*.rnd`), `Frames` | `attr.worker.tag.mistika = "true"` |
| `mistika-vr-render` | `vr -r <scene> -s/-e <frame range>` | chunks of 10 | `SceneFile` (`*.rnd`), `Frames` | `attr.worker.tag.mistikavr = "true"` |
| `mistika-workflows-render` | `workflows -r <scene> -s/-e <frame range>` | chunks of 10 | `SceneFile` (`*.rnd`), `Frames` | `attr.worker.tag.mistikaworkflows = "true"` |

Maya and Nuke each ship two variants: a **single-target** product (`-rl <layer>` /
`-X <writeNode>`) and a **whole-scene** product that omits that flag — Maya's
Expand All @@ -265,18 +272,33 @@ Nuke instead get multi-frame chunks (10 by default) natively, because both the
embedded `hython` script and `nuke -F` accept a frame range per invocation, so a
chunk amortizes DCC startup cost across several frames.

The three Mistika presets use `SQI_CHUNK_BOUNDS` the same way Maya and Blender
do, but batch by default (chunks of 10): every Mistika renderer takes `-s
START -e END`, so a chunk maps directly onto one invocation. All three render
`.rnd` job files. For a single movie-file render, submit `Frames` as `1`.

Each preset's `hostRequirements` gates it to workers advertising the matching
`attr.worker.tag.<name>` capability tag. `sqi-worker` auto-detects a standard
Maya/Nuke/Houdini/Blender install on `PATH`/in its usual install location and
advertises the matching tag (`maya`, plus a version variant like `maya-2025`)
with value `"true"` automatically at startup, with no configuration — see
[Capability
Maya/Nuke/Houdini/Blender/Mistika install on `PATH`/in its usual install
location and advertises the matching tag (`maya`, plus a version variant like
`maya-2025`) with value `"true"` automatically at startup, with no
configuration — see [Capability
auto-detection](worker-capabilities.md#capability-auto-detection-built-in-dcc-detectors).
That satisfies the `anyOf: ["true"]` requirement each of the six presets above
That satisfies the `anyOf: ["true"]` requirement each of the nine presets above
declares, so a worker with a standard DCC install matches these presets with
**zero per-worker configuration**. Run `sqi-worker capabilities` (or
`start --dry-run`) to confirm at a glance which DCCs were actually found.

**The Mistika detectors probe install locations only, never `PATH`.** The
binaries are named `mistika`, `vr` and `workflows`; the last two are generic
enough that a `PATH` probe would tag workers with no Mistika installed, and a
falsely tagged worker is handed render work it cannot run. The trade-off is
that detection and execution use different evidence: a worker is tagged
because Mistika is *installed* under the SGO layout, but the preset invokes
the bare command, so the binary must also be on the worker's `PATH`. If your
SGO install is not on `PATH`, either add it or duplicate the preset and give
`command:` an absolute path.

Manual tags are only needed when a DCC lives at a nonstandard install path (or
under a name/location the built-in detectors don't check) or for in-house
tools the built-in detectors don't cover. In those cases, add a `key=value`
Expand Down
2 changes: 1 addition & 1 deletion docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -1047,7 +1047,7 @@ clients/submitter/
qt/ Qt only, imported lazily (PySide6, falls back to PySide2)
hosts/{maya,houdini,nuke,blender}/ adapter + launch glue per host
tests/ pytest; tests/integration/ is env-gated (see below)
presets/sqi/ the eleven reference preset YAML fixtures (six DCC render
presets/sqi/ the fourteen reference preset YAML fixtures (nine DCC render
presets, five ffmpeg transcode presets), validated by
internal/product/sqipresets_test.go against a per-preset
`want` map of expected category/params/extensions —
Expand Down
5 changes: 4 additions & 1 deletion docs/preset-library.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,10 @@ copies show the **Update available** badge.
The official library ships `Rendering`-category presets — `maya-layer-render`,
`maya-scene-render`, `houdini-rop-render`, `nuke-write-render`,
`nuke-script-render`, `blender-batch-render` — as ready targets for the
[`sqi-submitter`](dcc-submitters.md) in-application submitters.
[`sqi-submitter`](dcc-submitters.md) in-application submitters, plus
`mistika-boutique-render`, `mistika-vr-render` and `mistika-workflows-render`,
which have no in-application submitter and are submitted from the web UI or
the API.
Each declares its parameters using the [parameter convention
contract](dcc-submitters.md#the-parameter-convention-contract) (`SceneFile`,
`Frames`, `OutputDir`, plus per-host extras) so submitter pre-fill works out of
Expand Down
9 changes: 5 additions & 4 deletions docs/products.md
Original file line number Diff line number Diff line change
Expand Up @@ -131,18 +131,19 @@ configuration (see [worker-capabilities.md](worker-capabilities.md)). Runs
`docker run --rm {{Param.Image}}`.

**A note on the DCC reference presets and auto-detection.** The
`presets/sqi/*.yaml` reference products (Maya, Nuke, Houdini, Blender — see
`presets/sqi/*.yaml` reference products (Maya, Nuke, Houdini, Blender, Mistika
Boutique/VR/Workflows — see
[docs/dcc-submitters.md](dcc-submitters.md#reference-presets)) gate the same
way: a `hostRequirements.attributes` entry requiring `attr.worker.tag.<app>`
with `anyOf: ["true"]`. `sqi-worker` auto-detects a standard install of each
of those four applications and advertises the matching tag (e.g. `maya`) with
of those applications and advertises the matching tag (e.g. `maya`) with
value `"true"` with no configuration — see [Capability
auto-detection](worker-capabilities.md#capability-auto-detection-built-in-dcc-detectors)
— which satisfies the `anyOf: ["true"]` match above directly, so a worker with
a standard install matches these four built-in gates with zero per-worker
a standard install matches these built-in gates with zero per-worker
configuration. `docker` has no built-in detector, so the `container` product
above still needs the manual `capability_tags: ["docker=true"]` entry.
Nonstandard install paths for Maya/Nuke/Houdini/Blender, or an in-house tool
Nonstandard install paths for Maya/Nuke/Houdini/Blender/Mistika, or an in-house tool
not covered by a built-in detector, can add a manual tag or a [custom
detector](worker-capabilities.md#writing-custom-detectors) instead.

Expand Down
9 changes: 5 additions & 4 deletions docs/release-runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -247,9 +247,10 @@ VERSION=0.2.0

## Step 6b: Publish reference and testing presets to the preset library

The presets authored in this repo — the eleven `presets/sqi/*.yaml` reference
presets (Maya layer + scene, Nuke write + script, Houdini ROP, Blender batch;
ffmpeg transcode, sequence-encode, and segment-transcode bash/PowerShell/EXPR)
The presets authored in this repo — the fourteen `presets/sqi/*.yaml` reference
presets (Maya layer + scene, Nuke write + script, Houdini ROP, Blender batch,
Mistika Boutique + VR + Workflows; ffmpeg transcode, sequence-encode, and
segment-transcode bash/PowerShell/EXPR)
and the four `presets/testing/*.yaml` smoke-test presets (test-render and
test-steps, each in bash + PowerShell) — are not distributed as part of the
GitHub release or PyPI packages; they are published to the separate
Expand Down Expand Up @@ -294,7 +295,7 @@ go run ./cmd/presetgen -presets presets/testing -subdir testing -out /path/to/sq

**Verify:**
- The published index (`https://uberware.github.io/sqi-presets/index.json`)
lists all eleven presets under `sqi/` and the four test presets under `testing/`
lists all fourteen presets under `sqi/` and the four test presets under `testing/`
with the correct `sha256`.
- In the sqi web UI's Preset Library page, each preset installs successfully
and its checksum verifies (no 422).
Expand Down
2 changes: 1 addition & 1 deletion docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -225,7 +225,7 @@ NATS can run embedded within `sqi-server` (simple mode) or as a separate cluster
- S3-compatible storage support (thin layer: derived type, root validation, path staging via operator sync tool)
- DCC submitter framework — in-application submitters for Maya, Houdini, Nuke, and Blender (the `sqi-submitter` Python package), built on the Python client
- Compute location registry and step-level affinity (native OpenJD `attr.worker.computelocation`)
- Chunk bounds (`SQI_CHUNK_BOUNDS` vendor extension) — expose each task chunk's frame start/end to the command line, used by the Maya and Blender reference presets
- Chunk bounds (`SQI_CHUNK_BOUNDS` vendor extension) — expose each task chunk's frame start/end to the command line, used by the Maya, Blender and Mistika reference presets
- Auto-retry and failure limits — per-task retry policy (max attempts, retry delay) and a job-level failure ceiling that auto-parks a job, resolved over four tiers (server → farm → queue → job) with per-task attempt history
- Cross-job dependencies — a submission may declare `depends_on` upstream jobs (same farm, across queues); dependents are held `blocked` until every upstream completes, then released (or canceled if an upstream fails)
- Testing job presets — ready-to-run `test-render`/`test-steps` presets (bash and PowerShell) published to the preset library for smoke-testing a farm
Expand Down
23 changes: 19 additions & 4 deletions docs/worker-capabilities.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,11 +112,12 @@ GPU-capable on macOS or Windows.

In addition to the hardware/OS tags above, `sqi-worker` runs a second,
declarative detection engine (`internal/worker/capabilities`) that looks for
installed creative applications — Maya, Nuke, Houdini, Blender — and
installed creative applications — Maya, Nuke, Houdini, Blender, Mistika
(Boutique/Ultima, VR, Workflows) and ffmpeg — and
advertises a tag with value `"true"` automatically, with no per-worker
configuration. This makes the software actually installed on a worker visible
without hand-editing its config, and it's enough on its own to satisfy the
`anyOf: ["true"]` gate the six shipped reference presets declare — a standard
`anyOf: ["true"]` gate the shipped reference presets declare — a standard
install matches those presets with zero configuration. Run
`sqi-worker capabilities` any time to see what was found. See
[`docs/dcc-submitters.md`](dcc-submitters.md#reference-presets) for how these
Expand Down Expand Up @@ -145,7 +146,7 @@ running worker is only picked up on the next restart.

### Built-in detectors

Four built-ins ship embedded in the worker binary, one YAML file per
Eight built-ins ship embedded in the worker binary, one YAML file per
application under
[`internal/worker/capabilities/builtins/`](https://github.com/uberware/sqi/tree/main/internal/worker/capabilities/builtins):

Expand All @@ -155,6 +156,20 @@ application under
| `houdini` | `exe hython` (any OS, if on `PATH`); Linux `path_glob /opt/hfs*/bin/houdini`; macOS `path_glob /Applications/Houdini/Houdini*/Frameworks/Houdini.framework`; Windows `registry HKLM\SOFTWARE\Side Effects Software\Houdini` | `(?:hfs\|Houdini)(?P<v>[0-9]+\.[0-9]+)` against the matched path |
| `nuke` | Linux `path_glob /usr/local/Nuke*/Nuke*`; macOS `path_glob /Applications/Nuke*/Nuke*.app`; Windows `registry HKLM\SOFTWARE\The Foundry\Nuke` | `Nuke([0-9]+\.[0-9]+)` against the matched path |
| `blender` | `exe blender` (any OS, if on `PATH`); macOS `path_glob /Applications/Blender.app`; Windows `registry HKLM\SOFTWARE\BlenderFoundation\Blender` | none (presence only — no `-<version>` tags) |
| `ffmpeg` | `exe ffmpeg` (any OS, if on `PATH`) — a PATH tool everywhere, so no install glob and no registry probe | none (presence only) |
| `mistika` | Linux `path_glob /home/mistika/SGO Apps/Mistika Ultima*/bin/mistika`; macOS `path_glob /Applications/SGO Apps/Mistika Boutique.app*/Contents/MacOS/mistika`; Windows `path_glob C:\Program Files\SGO Apps\Mistika Boutique*\bin\mistika.exe` | `Mistika (?:Ultima\|Boutique) ?(?P<v>[0-9]+\.[0-9]+)` against the matched path |
| `mistikavr` | Linux `path_glob /home/mistika/SGO Apps/Mistika VR*/bin/vr`; macOS `path_glob /Applications/SGO Apps/Mistika VR.app*/Contents/MacOS/vr`; Windows `path_glob C:\Program Files\SGO Apps\Mistika VR*\bin\vr.exe` | `Mistika VR ?(?P<v>[0-9]+\.[0-9]+)` against the matched path |
| `mistikaworkflows` | Linux `path_glob /home/mistika/SGO Apps/Mistika Workflows*/bin/workflows`; macOS `path_glob /Applications/SGO Apps/Mistika Workflows.app*/Contents/MacOS/workflows`; Windows `path_glob C:\Program Files\SGO Apps\Mistika Workflows*\bin\workflows.exe` | `Mistika Workflows ?(?P<v>[0-9]+\.[0-9]+)` against the matched path |

The three Mistika detectors are the only built-ins that probe Windows with a
`path_glob` rather than a `registry` key — SGO installs under a versioned
directory and registers no well-known key. They also deliberately carry **no
`exe` check**: their binaries are `mistika`, `vr` and `workflows`, and a bare
`PATH` probe for the last two would tag workers that have no Mistika at all.
Detection is therefore by install location only, while the presets still
invoke the bare command — so the binary must also be on the worker's `PATH`
for the job to run. The install layouts are taken from the corresponding
Smedge product definitions.

A `path_glob`/`exe` check's matched signal is a filesystem path, so version
extraction works there. A `registry` check's signal is the registry key
Expand Down Expand Up @@ -439,7 +454,7 @@ steps:
anyOf: ["true"]
```

This is the real syntax the six DCC reference products under `presets/sqi/`
This is the real syntax the nine DCC reference products under `presets/sqi/`
use — see [`docs/dcc-submitters.md`](dcc-submitters.md#reference-presets).
(`presets/sqi/` also ships `Transcoding`-category ffmpeg presets that gate on
`attr.worker.tag.ffmpeg` instead — see [`docs/preset-library.md`](preset-library.md#transcoding-reference-presets).)
Expand Down
17 changes: 10 additions & 7 deletions internal/openjd/exprlimits.go
Original file line number Diff line number Diff line change
Expand Up @@ -191,13 +191,16 @@ type ExprLimits struct {
// guard firing first, which is a derivation this file offered once and had to
// withdraw (see [ExprLimits.SubmissionMemoryBytes]).
//
// The FLOORS all come from measurement, not from taste. The six reference
// presets in presets/sqi/ -- the templates this repo itself ships -- were
// measured per dimension by binary search: they cost at most 15 expression
// positions, retain 0 let bytes, need at most 64 live bytes and at most 1
// operation in any single evaluation. Every floor below leaves at least 4x
// headroom over the worst of them (17x to 65536x in practice), so no operator
// can tighten a knob to a value that rejects sqi's own templates.
// The FLOORS all come from measurement, not from taste. The fourteen reference
// presets in presets/sqi/ -- the templates this repo itself ships -- are
// measured per dimension by binary search on every run, by
// TestExprLimits_FloorsAcceptReferencePresets: at most 31 expression positions,
// 1 retained byte, 390 live bytes and 15 operations in any single evaluation.
// Every one of those maxima is set by ffmpeg-segment-transcode-expr, the only
// preset doing real slice arithmetic; the render presets cost an order of
// magnitude less. Every floor below leaves at least 4x headroom over the worst
// of them (8x to 65536x in practice), so no operator can tighten a knob to a
// value that rejects sqi's own templates.
// TestExprLimits_FloorsAcceptReferencePresets re-measures that on every run
// rather than trusting this paragraph.
const (
Expand Down
2 changes: 1 addition & 1 deletion internal/openjd/exprlimits_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ func TestExprLimits_OrDefaults(t *testing.T) {
// asking the reader to trust a comment.
//
// A limit set absurdly low does not fail safe: it rejects legitimate templates
// at submit. The cheapest available definition of "legitimate" is the eleven
// at submit. The cheapest available definition of "legitimate" is the fourteen
// reference presets this repo itself ships (presets/sqi/*.yaml, the same files
// internal/product/sqipresets_test.go validates), so no floor may be tight
// enough to reject one of them.
Expand Down
7 changes: 4 additions & 3 deletions internal/presetgen/presetgen_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -21,16 +21,17 @@ func TestBuild(t *testing.T) {
if err != nil {
t.Fatalf("Build: %v", err)
}
if len(got) != 11 {
t.Fatalf("got %d presets, want 11", len(got))
if len(got) != 14 {
t.Fatalf("got %d presets, want 14", len(got))
}
// Sorted by name.
want := []string{
"blender-batch-render", "ffmpeg-segment-transcode-bash",
"ffmpeg-segment-transcode-expr",
"ffmpeg-segment-transcode-powershell", "ffmpeg-sequence-encode",
"ffmpeg-transcode", "houdini-rop-render", "maya-layer-render",
"maya-scene-render", "nuke-script-render", "nuke-write-render",
"maya-scene-render", "mistika-boutique-render", "mistika-vr-render",
"mistika-workflows-render", "nuke-script-render", "nuke-write-render",
}
for i, w := range want {
if got[i].Entry.Name != w {
Expand Down
18 changes: 18 additions & 0 deletions internal/product/sqipresets_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,24 @@ func TestSQIReferencePresets(t *testing.T) {
params: []string{"SceneFile", "Frames", "OutputPath"},
extensions: []string{"TASK_CHUNKING"},
},
// The three Mistika presets are the only shipped presets that declare
// SQI_CHUNK_BOUNDS: they invoke the renderer as `-s START -e END`, so
// they need a chunk's bounds rather than the chunk itself.
"mistika-boutique-render": {
category: "Rendering",
params: []string{"SceneFile", "Frames"},
extensions: []string{"TASK_CHUNKING", "SQI_CHUNK_BOUNDS"},
},
"mistika-vr-render": {
category: "Rendering",
params: []string{"SceneFile", "Frames"},
extensions: []string{"TASK_CHUNKING", "SQI_CHUNK_BOUNDS"},
},
"mistika-workflows-render": {
category: "Rendering",
params: []string{"SceneFile", "Frames"},
extensions: []string{"TASK_CHUNKING", "SQI_CHUNK_BOUNDS"},
},
"ffmpeg-transcode": {
category: "Transcoding",
params: []string{"SourceFile", "OutputFile", "VideoCodec", "Quality", "AudioCodec"},
Expand Down
Loading