Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
84 commits
Select commit Hold shift + click to select a range
45c0978
feat: add consent-based Jev discovery to Agent Match
Oct 24, 2026
966ba69
fix: enforce Agent Match output rules and freeze paired comparison gates
Oct 24, 2026
e3dd245
test: add private preview runner for frozen Agent Match comparison
Oct 24, 2026
d1b2e5c
fix: preserve type coverage without restoring unrelated candidates
Oct 24, 2026
1569915
test: preflight managed preview access before reserving paid attempts
Oct 24, 2026
f760110
fix: supply Agent Match JSON contract to the Lovable gateway
Oct 24, 2026
647d883
fix: share the bounded GPT-5.2 generation profile with production
Oct 24, 2026
ae55d69
test: measure paired Jev and generation in the same managed backend
Oct 24, 2026
241a5ac
fix: stream Agent Match results within the unchanged handler deadline
Oct 24, 2026
0f00a4b
test: isolate streaming transport phases without model inference
Oct 24, 2026
276dc7c
fix: align Seroval runtime and stream plugins at 1.7.2
Oct 24, 2026
227723f
test: freeze the validated streaming comparison profile and remove de…
Oct 24, 2026
c01176d
fix: ground match roles in documented package scope
Oct 24, 2026
3ac464e
test: diagnose managed TypeSafe failures without generator retries
Oct 24, 2026
2979962
test: classify provider transport failures without exposing messages
Oct 24, 2026
6613eb1
fix: reject redirects with managed-compatible manual transport
Oct 24, 2026
a5a19a4
test: prepare same-backend comparison without redundant local inference
Oct 24, 2026
84e7039
fix: send strict Agent Match schema to the generator
Oct 24, 2026
0a3e8e5
fix: preserve documented package outcomes and reconcile bounded compa…
Oct 24, 2026
3cd6a3d
fix: constrain generated package slugs to their documented types
Oct 24, 2026
fab6100
fix: bind verification scope and durably journal full comparison cohorts
Oct 24, 2026
a4dd61a
fix: share strict scope enums within provider limits without dropping…
Oct 24, 2026
581cb7a
fix: bound match client waits and reject unsupported recommendations
Oct 24, 2026
0621862
docs: record native zero-inference stream validation
Oct 24, 2026
577c2f2
test: honor owner-authorized seven dollar accumulated benchmark cap
Oct 24, 2026
f43dac6
fix: anchor match language and documented scope before recommending
Oct 24, 2026
8e49df7
fix: bind generated recommendations to verbatim catalog evidence
Oct 24, 2026
5231342
fix: label exact description evidence and prefer minimal match kits
Oct 24, 2026
833f0a4
fix: choose typed slug before copying its exact source evidence
Oct 24, 2026
e3c8b13
fix: evaluate matched scope with bounded low reasoning in both arms
Oct 24, 2026
a925b1b
fix: require strong documented fit without padding recommendation quotas
Oct 24, 2026
51ee75a
fix: reason over documented input objects and supplied capability gaps
Oct 24, 2026
ab3502a
fix: clarify payment connector scope and reuse common prompt prefixes
Oct 24, 2026
c001cef
docs: record rejected full Jev comparison and safe batch planner
Oct 24, 2026
d6e0bd4
fix: bound repeated Jev selection and preserve accumulated inference …
Oct 25, 2026
4fe993a
fix: retain exact Jev selections across observed repeat intervals
Oct 25, 2026
392ba8f
chore: integrate published runtime and package guidance before new ma…
Oct 25, 2026
d2ca780
chore: authorize cumulative USD10 comparison without resetting prior …
Oct 25, 2026
04cfb35
docs: record USD10 preflight and unchanged qualification contract
Oct 25, 2026
a6047c2
fix: keep knowledge-source capabilities out of transaction matching
Oct 25, 2026
fa174b5
Preserve rejected Agent Match costs and terminal evidence
Oct 25, 2026
b4e363f
Expose inference-free administrator cache continuity diagnostics
Oct 25, 2026
804ad95
Show bounded cache preflight receipts without touching inference jour…
Oct 25, 2026
db1ff1d
Initialize cache continuity identifiers inside Worker requests
Oct 25, 2026
42442fa
Verify selector reuse with bounded synthetic provider receipts and re…
Oct 25, 2026
3bb6628
Close rejected cohorts and retain managed cache instance-loss evidence
Oct 25, 2026
b500e0f
Persist scoped Jev selection evidence across managed Worker replacements
Oct 25, 2026
1e17cb7
Retain verified cross-instance cache reuse and strict probe validation
Oct 25, 2026
faf12be
Make cancellation fixture deterministic on the CI Node22 runtime
Oct 25, 2026
14ba21a
Require catalog counterevidence and reconcile verified no-start reserve
Oct 25, 2026
dc26919
Pin fresh v31 comparison to unchanged original cohort
Oct 25, 2026
1278e55
Integrate roadmap changes and authorize accumulated USD20 qualificati…
criptogus Oct 25, 2026
a440de3
Merge remote-tracking branch 'origin/main' into codex/jev-agent-match…
criptogus Oct 25, 2026
051a095
Pin USD20 comparison to original cohort and current roadmap source
criptogus Oct 25, 2026
e0e02f7
Avoid unsupported strict schema composition and retain failed-attempt…
criptogus Oct 25, 2026
6466a94
Allow validated frozen benchmark batches to be pasted without browser…
criptogus Oct 25, 2026
ba7014d
Buffer generator evidence once and exercise full SDK transport withou…
criptogus Oct 25, 2026
305416c
Separate unknown package detail from missing requested capabilities
criptogus Oct 25, 2026
480154f
Close failed generator cohorts and retain reconciled native charges
criptogus Oct 25, 2026
496b71a
Merge current main while preserving timeout and security boundaries
Oct 28, 2026
ec31dcb
Resume original GPT5.2 qualification under accumulated USD10 cap
Oct 28, 2026
6dc3d82
Pin v38 comparison to original cases and restricted budget
Oct 28, 2026
782b302
Record resumed qualification gates and verified workspace limit
Oct 28, 2026
7084a78
Close rejected v38 and require per-gap request and catalog evidence
Oct 28, 2026
84267bb
Work in progress
lovable-dev[bot] Oct 28, 2026
e8ba41b
Changes
lovable-dev[bot] Oct 28, 2026
b438714
Changes
lovable-dev[bot] Oct 28, 2026
803af6f
Changes
lovable-dev[bot] Oct 28, 2026
65b6c79
Changes
lovable-dev[bot] Oct 28, 2026
7c4633f
Changes
lovable-dev[bot] Oct 28, 2026
7e94d86
Changes
lovable-dev[bot] Oct 28, 2026
9945050
Changes
lovable-dev[bot] Oct 28, 2026
b955b95
Changes
lovable-dev[bot] Oct 28, 2026
568304a
Changes
lovable-dev[bot] Oct 28, 2026
e19d374
Synced pricing copy to GitHub
lovable-dev[bot] Oct 28, 2026
c7d9056
Changes
lovable-dev[bot] Oct 28, 2026
51dc9a8
Changes
lovable-dev[bot] Oct 28, 2026
0ec6ab0
Copied pricing files from GitHub
lovable-dev[bot] Oct 28, 2026
94505e6
Changes
lovable-dev[bot] Oct 29, 2026
43cdca5
Changes
lovable-dev[bot] Oct 29, 2026
26af965
Changes
lovable-dev[bot] Oct 29, 2026
61fe867
Changes
lovable-dev[bot] Oct 29, 2026
6ac5fac
Changes
lovable-dev[bot] Oct 29, 2026
721d81c
Separated Pro access by agent_id
lovable-dev[bot] Oct 29, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 56 additions & 0 deletions .agents/skills/validate-sas-release/SKILL.md

Large diffs are not rendered by default.

7 changes: 7 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,13 @@ AI_GATEWAY_BASE_URL=https://api.openai.com/v1
AI_GATEWAY_API_KEY=
AI_GATEWAY_MODEL=openai/gpt-4o-mini

# Optional Agent Match discovery. Server secret; never use a VITE_ prefix.
# Jev is a separate TypeSafe provider and never uses the Lovable AI gateway.
TYPESAFE_API_KEY=
# off | shadow | filter. Default shadow: measure without pruning generator context.
# Promote to filter only after paired end-to-end quality/cost evaluation.
JEV_AGENT_MATCH_MODE=shadow

# --- Trust Score: release signing (Phase 2 + Phase 6) ------------------------
# Generate locally with:
# openssl genpkey -algorithm ed25519 -out priv.pem
Expand Down
5 changes: 5 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Technical decisions

- Pro entitlements use explicit price lookup keys and all matching subscriptions in the current payments environment through `subscription-entitlements.ts` and `auth/pro-subscription.server.ts`; standalone Agent Pass is separate. Why: a paid seat or newest subscription must not grant or hide Pro access.

- `src/integrations/supabase/database.ts` merges hand-written schema contracts (`optional-schema.ts`, `platform-schema.ts`) only for entries missing from the generated `types.ts`; generated entries always win. Why: intersecting duplicate table definitions with different `Relationships` breaks supabase-js embedded-resource type parsing (embedded selects like `packages(slug)` fail to resolve).
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ Component artifacts are versioned and released independently via git tags:
- Bind approval writes to the evaluated version and confirm the returned reviewer/status; reject errors and no-op writes.
- Keep failed server responses from being reported as successful review decisions.

## [0.6.4] — 2026-10-23 — *Genesis* (alpha) — Maintenance release
## [0.6.4] — 2026-10-23 — *Genesis* (alpha) — Optional Jev selection for Agent Match

### Changed

Expand Down
314 changes: 285 additions & 29 deletions bun.lock

Large diffs are not rendered by default.

475 changes: 475 additions & 0 deletions docs/AGENT-MATCH-PAIRED-EVALUATION.md

Large diffs are not rendered by default.

2 changes: 2 additions & 0 deletions docs/ROADMAP-STATUS.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,3 +80,5 @@ disabled, and the GPT-5.4 proposal was not executed.
## Agent Match qualification stopped by owner

On 2026-10-25 the owner chose to keep Jev disabled after two GPT-5.2 baseline attempts failed primary NF-e gap review. Neither attempt constitutes a passing paired cohort or evidence of a Jev-arm defect. The separate GPT-5.4 proposal remains unpushed and unexecuted. No further paid Agent Match calls are planned. Historical charges and unresolved reservations remain preserved under the previously authorized cumulative US$20 ceiling. Draft PR #77 must not be promoted as qualified.

On 2026-10-28 the owner explicitly requested resuming the original GPT-5.2 Agent Match comparison under an accumulated USD10 limit. This supersedes the prior stop decision for the unpublished candidate only; all historical charges/reservations and failed cohorts remain retained. No GPT-5.4 migration is authorized. Jev filtering stays disabled in production pending complete original qualification.
55 changes: 55 additions & 0 deletions docs/adr/20261024-jev-agent-match.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Optional Jev discovery in Agent Match

Status: implemented on the draft branch; filtered rollout rejected by the complete v26 paired evaluation. No production publication.

The v7 candidate replaces process-local reuse on the authenticated production/benchmark paths with service-only Supabase probability evidence. The original scope, exact eligible catalog, provider key, model and protocol remain in the opaque cache identity; no briefing text, credentials or generator output is stored. The private RPCs keep a fixed thirty-minute expiry and a global 64-entry bound. Duplicate writers cannot replace the first evidence or extend expiry. Missing/malformed/expired cache data is a miss; database calls are bounded at 500 ms and share the selector's original three-second total deadline (maximum five seconds when explicitly configured). Provider requests still have no automatic retry. This candidate is not qualified or published.

This change follows an actual managed-instance replacement after 426,396 ms that lost v6's in-memory evidence while the credential partition remained identical. A short repeat hit first; then the same probe required a second real Jev request despite the thirty-minute expiry. The v7 adapter's independent-client HTTP regressions and twelve native SQL rollback checks passed, including real anon/authenticated denials, actual service-role access, unchanged expiry, strict evidence validation and the global bound. The additive private migration is installed but unused by the production0.7.1 app. Full qualification and public activation remain blocked until all original paired quality, conservative total-cost and client-p95 gates pass within the owner's cumulative USD10 cap.

Agent Match currently supplies up to 800 approved, published package metadata records to its generator. A separate TypeSafe client now scores task relevance before that call. It does not generate the kit, approve a package, change model compatibility or weaken any existing output validation.

## Data and authority

`semantic_selection` defaults to false. The user must explicitly consent to sending their brief, vertical and public candidate descriptions to TypeSafe. `TYPESAFE_API_KEY` is a server secret; it has no VITE prefix, browser use or fallback to the Lovable gateway. Existing publication/status filters run before selection. Generator output remains restricted to candidate slugs and correct package types, with the original count/score and MCP checks preserved.

The fixed provider is `https://api.typesafe.ai/v1/systemone`, model `jev-1.14.0`, protocol `agent-match-v4`. Individual questions name the capability and slug explicitly; numeric-index-only references caused unreliable results in the initial live pilot. Matching asks about intended jobs, rather than whether all execution records are already supplied in the brief. Input/output/schema bounds, exact answer sets and actual model are validated.

## Operation

- Set `TYPESAFE_API_KEY` securely on the server. Do not copy the local Keychain credential into source, a VITE variable, logs or a PR.
- `JEV_AGENT_MATCH_MODE=off` disables calls. Missing key, invalid mode or absent consent also makes zero calls.
- `shadow` is the default. It records scores/selection metrics while forwarding the complete original catalog. It adds bounded evaluation cost and does not claim token savings.
- `filter` is an explicit operator setting for a qualified rollout, not a client parameter. Never activate merely because the smoke test passes. Compare full-catalog and filtered kits on the same representative tasks, measuring final quality and total model costs first.
- Score the full catalog, in batches of at most 40 candidates with at most four calls in parallel. There is no lexical shortlist that silently discards unexamined candidates. The existing 800-row catalog limit bounds calls at 20. Global deadline: 3 seconds (hard maximum 5 seconds in test/options); no automatic retries or redirects. Each request and response has a byte cap.
- Filtering retains every candidate above 0.1 relevance, including ambiguous candidates. It requires at least one score >=0.8 and per-type coverage of requested counts plus two alternatives (limited by actual availability). These are conservative experimental selection settings, not calibrated correctness guarantees. When a type lacks the minimum, restore every candidate of that type, retaining original order; no confident match still restores the entire catalog. This v4 change preserves the same minimum counts and uncertainty thresholds. No evidence means full-catalog recovery.
- Any malformed response, wrong model, missing/extra score, HTTP/network/deadline error or failed batch restores the whole original catalog. Partial successful batches cannot prune. Unknown usage after a failed request is explicitly marked incomplete rather than reported as free.
- The current draft candidate reuses complete probabilities for identical credential-scoped inputs for at most thirty minutes, fixed from insertion, in a 64-entry process cache. Every lookup binds the fresh eligible catalog and exact semantic inputs; no raw briefs or credentials are stored. Worker restarts/cold instances can miss. Candidate pruning runs only against the current catalog. Do not interpret omitted candidates as proof that the marketplace lacks a capability.

The response and structured server metrics include the actual selected/forwarded counts, Jev usage, protocol/model, status/reason and latency, plus generator input/output usage when supplied by its provider. They exclude briefs, package text, API keys and raw provider error bodies. Capture the metrics in the deployment's existing log system; no claim of durable database telemetry is made.

## Validation and limitations

The normal project batch tests consent/configuration no-call gates, provider contract, shadow behavior, conservative recovery, model drift, malformed answers/usage, deadline, response bounds, full-catalog order, concurrency and failed-batch recovery. A separate explicit live runner uses the production adapter and frozen synthetic fixtures; it never calls the generative gateway or application POST endpoint.

Live protocol histories on 2026-10-24:

| Protocol | Smoke result | Input/output tokens | Outcome |
| -------------- | ------------ | ------------------- | ------------------------------------------------------------ |
| agent-match-v1 | 6/8 | 25,260 / 2,976 | Retained as failed evidence |
| agent-match-v2 | 7/8 | 29,188 / 2,976 | Retained as failed evidence |
| agent-match-v3 | 8/8 | 36,580 / 2,976 | Discovery smoke passed; not end-to-end quality qualification |

The 0.8 expected-candidate check and original eight briefs stayed fixed. No fixture or threshold was weakened. Records live in `artifacts/jev-agent-match-live*-20261024/results.json`; the earlier independent Choice pilot is a different protocol and cannot qualify this one.

The owner authorized up to US$5 accumulated in native Lovable AI inference. Both backend credentials are configured and verified through readiness probes. The real paired v16 prefix completed 14/72 observations: eight primary passes and six failures, with p95 regression. Attempt 15 remained unknown and was stopped without replay. This is rejected, incomplete qualification; production filtering remains unpromoted. See the paired evaluation document for preserved billing, reservations, fixes and the next frozen profile. The 21 skills' registered artifacts, adversarial evidence and restricted supported runtime remain untouched.

Full-catalog selection evidence: the initial v3 round (36 observations, 612 real calls) forwarded all 665 candidates in 32 observations. The v4 repair restores only undersupplied types, while preserving all their alternatives. A separate 36-observation real round forwarded an average 328.39 candidates; six absent-capability observations correctly retained the entire catalog. Every frozen required slug was retained. Both rounds are preserved and neither qualifies generator quality, total cost or latency. The owner expanded the accumulated Lovable inference cap to US$7. Failed v18/v19 prefixes and the invalid v20 source quotation remain preserved and rejected; v21 labels literal evidence separately from the package name and retains exact validation. Native billed upper bounds and unknown reservations remain deducted across rounds. Provider caching applies to both arms and can make the full catalog cheaper than filtering; reduced forwarded tokens alone cannot qualify rollout.

The complete frozen v26 cohort (72 outputs, 12 cases × three repeats × two arms) passed all primary quality dimensions with zero high/critical failures. It failed the unchanged efficiency gates: conservative total cost baseline US$0.8725 versus Jev US$1.623483414 (selection included), actual client p95 23,805 ms versus 28,230 ms. Input tokens decreased 43.58%, while observed cached input was approximately 94% for baseline and 36% for Jev. Do not enable filtering or publish this profile. See `docs/AGENT-MATCH-PAIRED-EVALUATION.md` and the retained `run-live-v26/comparison-072.json` for bindings and billing intervals. The accumulated US$7 cap remains respected: billed upper plus unknown holds US$6.56241575. No further inference was started.

V27 candidate corrections are retained on the draft branch only: exact five-minute credential-partitioned probability reuse, compact shared relevance policy with complete per-question public candidate metadata, cache provenance checks, and a server-side accumulated-allowance ceiling. Anonymous requests bypass reuse; no generator answers or primary approvals are cached. The selector protocol changes to v5 and thus requires fresh complete qualification. Deterministic regression/byte measurements and one five-candidate TypeSafe contract probe are not economic or quality approval. The new 72-generation comparison was not started: remaining native allowance is US$0.43758425, versus US$2.38 reference cohort native upper bound. No additional Lovable inference or builder calls, no release, and no weakened gates. See the paired-evaluation document for evidence and limitations.

The v28 candidate corrects a second cache-window defect: every one of the 24 repeated-selection intervals in the original sequential v26 schedule has an intervening-call latency lower bound above five minutes (371,917–447,318 ms). Therefore the v27 five-minute cache cannot stabilize those repeats even in a warm worker; reconciliation/UI pauses can only lengthen those bounds. The candidate uses a fixed thirty-minute expiry with the same 64-entry, exact-input, credential and provider-key boundaries; reads never extend lifetime, and eligibility is fetched before each production lookup. Model judgments, authority, approval thresholds and the original 72-case schedule remain unchanged. Protocol v6 and new source hashes require fresh complete qualification. The seven-minute regression failed with two calls before correction and passed with one afterward; it also proves expiry at thirty minutes. This is mechanism evidence, not a cache-hit guarantee or cost/p95 approval. No new Lovable inference or publication is authorized by these results.


Owner authorization on 2026-10-25 raises only the accumulated Lovable inference ceiling to USD10. Prior native billed upper USD5.48125 and unknown holds USD1.08116575 remain deducted. The new v29 plan retains the exact original cases, catalog, schedule, generator and criteria; no qualification or production activation is implied. Historical USD7 summaries remain immutable.
Loading