Build cross-tool orchestration and failure fixture suites

[szmyty]

Outcome

Create the suite-level conformance fixtures required to prove that flow can plan, execute, validate, explain, interrupt, retry, and resume work across independently released renderflow, optiflow, and aniflow providers.

Flow owns orchestration fixtures and expected cross-tool state transitions. Raw format corpora remain in the provider repositories.

Motivation

Apache Tika-style parser corpora are valuable because they turn rare format behavior, malformed inputs, and regressions into durable tests. This repository needs the equivalent for its own product boundary.

The corpus must be versioned, deterministic, redistribution-safe, inspectable, and generator-first. It must not become an unreviewed dump of large or copyrighted binaries, and its coverage manifest must never claim that finite fixtures prove support for every possible file.

Cross-suite rules

• Keep fixture generation and assertions in the owning repository.
• Use public CLI/API and versioned artifacts for cross-tool tests; never import sibling source trees.
• Prefer small source recipes plus deterministic generators over committed large binaries.
• Record seed, generator/tool version, parameters, expected digest, media type, validity class, license/provenance, and expected behavior.
• Partition fixtures into valid, boundary, malformed, unsupported, and resource-stress classes.
• Malformed fixtures must have intentional corruption recipes and named expected diagnostics.
• Resource-stress cases must be safely bounded in normal CI; expensive/nightly tiers must be explicit.
• Never include secrets, personal data, proprietary samples, malware, weaponized polyglots, or uncontrolled decompression bombs.
• Fuzz-found regressions must be minimized, assigned a stable ID, documented, and promoted into the deterministic corpus.
• Tests must validate typed outcomes, artifacts, digests, cleanup, and source immutability—not merely process exit or file existence.
• Keep generated files reproducible and add a drift check; CI must never rewrite canonical fixtures.

Required fixture families

Representative workflows

• Static publication/comic: immutable source → renderflow artifacts → optiflow inspection/optimization opportunity → validation and ship-readiness evidence.
• Temporal release: immutable time-based source → aniflow temporal processing/analysis → renderflow derivatives → optiflow inspection where semantically appropriate.
• Minimal single-provider and multi-provider graphs to isolate orchestration behavior.

Contract and compatibility cases

• Compatible, unavailable, missing, stale, future, and mismatched provider capability/contract versions.
• Valid and invalid provider locks, manifests, execution reports, provenance, digests, output declarations, and artifact sets.
• Partial outputs, invalid outputs, extra outputs, missing outputs, path escapes, symlinks, conflicting artifact identity, and provider success with failed validation.
• Upgrade/compatibility fixtures for every released Flow run/checkpoint/plan schema that policy promises to consume.

State-machine and authority cases

• Cancellation before start, during a provider, and between steps.
• Timeout, nonzero exit, signal termination, output-limit failure, host interruption, and restart.
• Resume after completed work, interrupted work, invalid checkpoint, changed inputs, changed provider lock, and changed plan.
• Retryable versus terminal failure; idempotency; no duplicate consequential side effects.
• Explicit refusal when mutation, upload, publication, signing, or another consequential action lacks authority.
• Redacted diagnostics and explanations that identify the failing boundary without leaking argv, environment, secrets, or private content.

Provider-fixture boundary

• Consume small immutable releases or pinned test artifacts from provider repositories where practical.
• Supply hermetic fake providers that implement public contracts for orchestration-only failures.
• Never reproduce renderflow parsing, optiflow inspection, or aniflow temporal algorithms inside flow.
• Verify at least one clean-room run using independently built/released provider artifacts rather than sibling workspaces.

Deliverables

• Stable workflow/fixture IDs and a machine-readable scenario manifest.
• Expected state-transition traces, plans, locks, reports, artifacts, digests, diagnostics, and explanations.
• Deterministic hermetic provider kit for orchestration failures.
• PR and scheduled/release tiers with resource budgets.
• Compatibility matrix covering supported provider and Flow contract versions.
• A contribution path for promoting production regressions into minimized scenarios.

Acceptance criteria

• Static-publication and temporal-release positive fixtures pass end to end.
• Negative fixtures cover unavailable capability, partial/invalid output, provider-version mismatch, corrupt artifact, and insufficient validation evidence.
• Flow never accepts provider exit or file existence as proof of success.
• Interruption and resume preserve completed work without duplicating side effects.
• Changed inputs, plans, provider locks, or artifacts invalidate unsafe resume.
• Authority boundaries are explicit and tested for consequential actions.
• Explanations are typed, actionable, and redacted.
• Provider repositories remain independently owned and source-decoupled.
• Clean-room tests use immutable provider artifacts.
• Fixture generation and expected evidence are deterministic and drift-checked.
• Current validation, contract, package, documentation, and release checks remain green.

Dependencies and scheduling

• This is a bounded child of suite epic [EPIC] Finish the flow suite and prove production-ready orchestration #11 and should be linked there.
• Reconcile with flow Implement a versioned cross-holon orchestration vertical slice #3 and [Publication] Orchestrate exhaustive comic finalization and multilingual release workflows #10: the corpus supplies reusable evidence and must not become a competing orchestration implementation epic.
• Provider-local corpus issues are dependencies for real raw-file coverage; hermetic fake-provider work can proceed independently.
• Complete before the final post-roadmap audit so that audit can verify behavior from repeatable evidence.

Non-goals

• Owning provider algorithms or raw-format corpora.
• Source-tree imports across repositories.
• Performing real consequential publication/signing in ordinary CI.
• Treating one happy-path demo as suite completeness.