Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .agents/skills/validate-sas-release/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,3 +84,7 @@ For Brazil source-state candidates, reuse the existing compiled-playbook batch,
After a complete four-playbook cohort passes primary review and strict reconstruction, reuse `sync:content --new-version` to prepare exact draft archives, compare all four archive digests to the frozen plan, and dry-run the atomic import with rollback before applying it. Retain a fresh native `registry.json` snapshot with immutable package/version IDs, source/archive SHA256, draft ownership and zero price. Run `node --experimental-strip-types scripts/prepare-solution-registry.mjs EVIDENCE_DIR NEW_SQL`: it independently reruns the strict report, rejects stale/incomplete/source/model bindings and prepares only qualifying model-scoped adversarial/compatibility records. It never changes publication/review flags. Validate SQL in rollback, then use the authenticated admin review UI and verify actual persisted/public versions afterward. Synthetic binding tests cover partial/duplicate/stale/public/broad-model/failed-primary reports; they are not live qualification.

After authenticated four-playbook publication, run `node --experimental-strip-types scripts/check-solution-publication.mjs EVIDENCE_DIR HTTPS_ORIGIN NEW_REPORT`. It rejects empty/stale targets, checks all public versions/source hashes and exclusive recorded runtimes, downloads each public ZIP and verifies every frozen file hash plus the complete artifact digest. It also verifies any configured detached ZIP signature; absence of signing configuration is recorded rather than represented as a verified signature. No model calls, writes or automatic retries. Re-run the existing21skill publication verifier and `check:security` after a public metadata change. The public package endpoint must select the registry's exact latest_version, and newly exposed recorded compatibility must match current source/archive, actual reviewed outcomes, severity counts/scores, protocol and version; never broaden a claim from a table row alone.

For model-scoped marketplace presentation, reuse the private `loadPublicEvaluationCompatibility` loader for both API metadata and detail pages; independent renderers must not substitute an empty declared array or a different version. Packaged execution guides require a supported runtime and distinguish source-state preparation from the separate local-model runner. Keep the complete evaluated ZIP, signature links, exact environment and source-upload consent visible; bare prompts, generic MCP installation and historical versions inherit no qualification. Run the real SSR regressions in `tests/packaged-execution-ui.test.ts` through the normal batch, then inspect the actual public page/compatibility counts. A missing runtime must block installation cleanly rather than dereferencing an empty array. Candidate UI validation passed333tests and12checks, including unchanged RLS/anonymous-access gates; no model/package contents changed.

Reuse `node scripts/check-public-version.mjs EXPECTED_VERSION CANONICAL_HTTPS_ORIGIN ALIAS_HTTPS_ORIGIN NEW_REPORT` after deployment. It makes exactly two anonymous reads without retries, requires the exact canonical version, and accepts an alias redirect only to that explicit canonical version endpoint. Foreign origins, added query parameters, stale versions and unknown observations fail. This exception is specific to the owned publication alias; model/source-upload transports retain their no-redirect policy. Actual0.7.0 verification passed2/2: the platform alias returned302to the canonical version endpoint, which returned200/version0.7.0. Earlier strict no-redirect probes remain preserved.
6 changes: 6 additions & 0 deletions docs/ADVERSARIAL-BUSINESS-EVALUATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -405,3 +405,9 @@ All four0.1.31 versions were approved through the existing authenticated Chrome
Public verification then caught a metadata gap: the portable archives were current, but `/api/public/packages/:slug` returned the version's empty declared compatibility instead of the recorded qualification. `check-solution-publication.mjs` reproduced0/4 before the fix. The API now pins its source query to `packages.latest_version` and adds recorded compatibility only after checking actual current-version run/outcomes, primary decisions, zero high/critical failures, exact weighted/count calculations and source/archive/protocol bindings. It exposes only that evaluated runtime when adding evidence; existing declarations without a usable recorded run remain declarations rather than new qualification. No immutable package/source/archive was rewritten.

The API helper passed2tests with13negative mutations; real current database fields mapped4/4to the expected exact-runtime claims in an offline helper probe (not a deployed HTTP proof). The complete batch passed331tests and all12checks including RLS/anonymous-access coverage, with unchanged allowlists (`artifacts/solution-public-api-project-security.json`). Deploy this metadata fix, then require the four-package public verifier to pass against real HTTP and exact ZIP files; retain the pre-fix failure and all original model histories.

Platform0.7.0 (PR75, commit034beaa7) was validated after the version bot (331tests/12checks), matched the native Lovable source and deployed directly without any builder message or inference. Production anonymous checks passed4/4source-state packages and21/21individual skills. All136frozen archive files and all4configured Ed25519 ZIP signatures verified; post-deployment RLS/anonymous-access checks passed across92tables without allowlist changes. Actual deployment/version/public/security receipts are retained in `artifacts/release-source-state-solutions`.

Visual verification then caught that marketplace pages still used declared compatibility only, displayed0runtimes and offered generic prompt/MCP instructions for packaged execution. The UI correction shares the same private source-bound compatibility loader with the API, selects the exact current version, recommends the complete evaluated ZIP/profile and explicit source-upload consent, and does not imply that prompt copying, arbitrary clients or historical versions are covered by the current evaluation. It also handles missing-runtime install preflight without a crash. Actual React SSR tests verify bundle/consent/preview output and reject partial/unsupported runtime declarations; all333tests and12project/security checks passed. These presentation changes do not modify immutable0.1.31archives or their original72case qualification. Deployment and visual/public verification of this follow-up remain pending.

The platform alias now redirects302to the canonical public version endpoint. Earlier probes used `redirect:error` and correctly recorded that redirect as a nonpassing direct response; no deployment was restarted. The small reusable `check-public-version.mjs` verifies the exact canonical version and permits only the explicit owned alias destination, never foreign origins or query-bearing redirects. Its2regressions passed and the real0.7.0batch passed2/2. Model/source transport rules and the original qualification criteria remain unchanged.
15 changes: 12 additions & 3 deletions docs/PLAYBOOK-RUNTIME-QUALIFICATION.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,17 @@
# Solution playbook runtime qualification

The four standalone `solution-*` playbooks remain unqualified. Their solution
pages and the approval of 21 individual skills do not approve execution of these
compiled workflows. No production publication is authorized by this document.
The four standalone `solution-*` playbooks qualified as immutable version0.1.31
under their exact source-state model/runtime profiles and were published through
authenticated administrator review. All72original cases passed primary source
review and strict parser/provider/stage reconstruction, with zero high/critical
failures. Publication does not transfer qualification to bare prompts, other
models, MCP transport alone or external financial/sending/security actions.

Current scope is Jev1.14.0 and each packaged Node24.22.0/arm64 protocol; Brazil also
pins Python3.12.15/Expat2.7.4. The21individual skills retain their separate packaged
local-model qualification. See `ADVERSARIAL-BUSINESS-EVALUATION.md` for current
proofs. The investigations below preserve historical candidate/failed protocols
and never supersede the original approval criteria.

The gateway CLI previously dropped `RunContext.inputs` and `RunContext.steps`.
The real CLI regression reproduced that omission and acceptance of both empty and
Expand Down
24 changes: 21 additions & 3 deletions docs/adr/20261025-finance-source-state.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
# Finance preparation uses typed state and source interpretation

Status: implemented candidate; qualification and publication pending.
Status: implemented; exact source-state profiles qualified and published at0.1.31.
The initial decision below is historical; see the final-scope note at the end.

Repeated real compiled-graph pilots preserved correct arithmetic but confused
budget approval, unmatched-entry ownership and authority to post/pay/sign off.
Expand All @@ -20,15 +21,18 @@ No generative model drafts the four finance stage outputs. This is an actual
execution-protocol change, not output repair, a replay or a mock substitute for a
model benchmark. Finance qualification is scoped to `jev-1.14.0`, the pinned Node
host and packaged finance-source-state protocol. It makes no Gemma quality claim.
The other three playbooks continue using the pinned local generator. Agent Match's
At the initial decision, the other three playbooks continued using the pinned
local generator; their later source-state migrations supersede that arrangement. Agent Match's
same-generator comparison and financial cap are unaffected.

The frozen four-subject/72-case contract, original severities, expectations,
thresholds, primary trace reviews and zero high/critical bar remain unchanged.
Finance evidence must contain a real source-interpretation response with model and
usage plus four actual accounting-tool receipts. The strict reporter rejects any
unexpected finance generation and recomputes each stage from the validated source
evidence. Other playbooks still require complete raw generation transports. No
evidence. At that initial stage, other playbooks still required complete raw generation
transports. Their final source-state protocols instead require actual interpreter
receipts and independently reconstructed tool outputs. No
prior qualification transfers to this protocol; all 72 cases remain required.

Authenticated external controller approval is a separate future integration. Until
Expand All @@ -40,3 +44,17 @@ human gate. It cannot claim that the business month is signed off.
The complete-output v12 run retained 19 real cases before an operator boundary stop. Accounting values and pending approval were correct in all 18 finance cases, but seven original baseline response requirements failed, including critical explicit-refusal omissions. Supplier generation also miscomputed landed cost. These failures remain preserved and block publication.

Finance v2 adds one bounded TypeSafe classification of locally sanitized attachment context, explicitly separate from original accounting sources. The selected safety response cannot authorize actions or change arithmetic. The raw model distribution remains in the provider evidence and all four actual preparation outputs retain the response. Sensitive attachment values and destinations are omitted before remote interpretation; unresolved detected leaks reject upload. This is a response-boundary correction, not a qualification claim. It requires a new package version, fresh source-bound run and unchanged case/scorer gates.

## Final qualified scope and presentation

All four0.1.31 graphs now use source-state preparation: finance-v3, supplier-v1,
security-v1 and Brazil-v1, pinned to Jev1.14.0/Node24.22.0/arm64. Brazil additionally
pins Python3.12.15/Expat2.7.4. The complete72case cohort used108actual TypeSafe
requests and zero generative requests; the MLX server was harness-only. No
generative-model or live-action qualification is inherited. All original cases,
thresholds, primary decisions and immutable archive/source bindings remain.

The marketplace must present the complete bundle and recorded runtime, rather
than treating a copied prompt, arbitrary MCP client or historical version as
that evaluated execution. API and page compatibility use the same bound evidence
loader. External authority remains pending independently of package approval.
77 changes: 77 additions & 0 deletions scripts/check-public-version.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
/** Two anonymous version reads, allowing only the explicit owned canonical redirect. */
import { existsSync, writeFileSync } from "node:fs";
import { resolve } from "node:path";
import { pathToFileURL } from "node:url";
export function versionResults(expected, canonical, alias, observations) {
const primary = observations[0],
secondary = observations[1];
const primaryPassed = primary?.status === 200 && primary.data?.version === expected;
const direct = secondary?.status === 200 && secondary.data?.version === expected;
let canonicalRedirect = false;
try {
canonicalRedirect =
[301, 302, 303, 307, 308].includes(secondary?.status) &&
new URL(secondary.location, alias).href === `${canonical}/api/public/version`;
} catch {}
return [
{ origin: canonical, ...primary, passed: primaryPassed },
{
origin: alias,
status: secondary?.status ?? null,
version: secondary?.data?.version ?? null,
error: secondary?.error ?? null,
canonical_redirect: canonicalRedirect,
passed: primaryPassed && (direct || canonicalRedirect),
},
];
}
async function main() {
const [expected, canonical, alias, output] = process.argv.slice(2);
if (
process.argv.length !== 6 ||
!/^\d+\.\d+\.\d+$/.test(expected ?? "") ||
!output ||
existsSync(output) ||
canonical === alias
)
throw new Error("Supply EXPECTED_VERSION CANONICAL_HTTPS_ORIGIN ALIAS_HTTPS_ORIGIN NEW_REPORT");
for (const value of [canonical, alias]) {
const u = new URL(value);
if (u.protocol !== "https:" || u.origin !== value || u.username || u.password)
throw new Error("Explicit credential-free HTTPS origins required");
}
const observations = await Promise.all(
[canonical, alias].map(async (origin) => {
try {
const r = await fetch(origin + "/api/public/version", {
redirect: "manual",
signal: AbortSignal.timeout(15000),
});
return {
status: r.status,
...(r.status === 200
? { data: await r.json() }
: { location: r.headers.get("location") }),
};
} catch (error) {
return { status: null, error: String(error) };
}
}),
);
const results = versionResults(expected, canonical, alias, observations);
const report = {
checked_at: new Date().toISOString(),
expected_version: expected,
total: 2,
passed: results.filter((r) => r.passed).length,
model_calls: 0,
production_writes: 0,
automatic_retries: 0,
results,
};
writeFileSync(output, JSON.stringify(report, null, 2) + "\n", { flag: "wx", mode: 0o600 });
console.log(JSON.stringify({ total: 2, passed: report.passed, report: output }));
process.exitCode = report.passed === 2 ? 0 : 1;
}
if (process.argv[1] && import.meta.url === pathToFileURL(resolve(process.argv[1])).href)
await main();
25 changes: 9 additions & 16 deletions src/components/marketplace/PackageGallery.tsx
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { useState } from "react";
import { packageInstallPreview, packagedExecution } from "@/lib/marketplace/package-execution";
import { FileCode2, Play, Terminal } from "lucide-react";
import type { LucideIcon } from "lucide-react";
import type { Package } from "@/data/packages";
Expand All @@ -25,31 +26,23 @@ export function PackageGallery({ pkg, gatewayUrl }: { pkg: Package; gatewayUrl:
const frames: Frame[] = [
{
id: "install",
label: "Install",
label: packagedExecution(pkg) ? "Evaluated bundle" : "Install",
icon: Terminal,
filename: "terminal",
tone: "terminal",
body: [
`$ npx super-agent install ${pkg.id}`,
"",
`→ resolving ${pkg.id}@${pkg.latest}`,
`→ runtimes ......... ${pkg.compatibility.map((c) => c.runtime).join(", ") || "any MCP client"}`,
`→ scopes ........... ${pkg.scopes.join(", ") || "none"}`,
`→ size ............. ${pkg.size}`,
"",
`✓ ${pkg.name} is available to your agent`,
"",
"# or point any MCP client at the gateway:",
`# ${gatewayUrl}`,
].join("\n"),
body: packageInstallPreview(pkg, gatewayUrl),
},
...(pkg.systemPrompt
? [
{
id: "prompt",
label: "System prompt",
label:
packagedExecution(pkg)?.kind === "source-state" ? "Workflow source" : "System prompt",
icon: FileCode2,
filename: `${pkg.id}.system-prompt.md`,
filename:
packagedExecution(pkg)?.kind === "source-state"
? `${pkg.id}.workflow.yaml`
: `${pkg.id}.system-prompt.md`,
tone: "editor" as const,
body: pkg.systemPrompt.split("\n").slice(0, 26).join("\n"),
},
Expand Down
Loading
Loading