Mirrored from fullsend-ai#2294 for Phase 2 QF integration testing
Problem
When running fullsend run twice against the same gateway (e.g. during local testing), the second run fails at the "Ensuring provider" step with:
provider create "github" failed: Error: × status: AlreadyExists, message: "provider already exists"
EnsureProvider in internal/sandbox/sandbox.go calls openshell provider create and treats any error as a hard failure. There is no handling for the AlreadyExists case — no silent success, no delete-and-recreate fallback, and no update path. The function comment says "creates or updates a provider" but only the create path is implemented.
The current workaround is to manually delete providers between runs (openshell provider delete github jira), which makes iterative local testing painful. Any failed or interrupted run leaves stale providers that block the next run.
Expected Behavior
EnsureProvider should be idempotent, consistent with the codebase's conventions:
- The
Provider interface requires Provision() implementations to be idempotent
EnsureGateway() is documented as idempotent in ADR 0030- The "Ensure" naming convention implies idempotent semantics
Possible Approaches
- Succeed silently on AlreadyExists — if the provider is already configured, treat it as success (true "Ensure" semantics)
- Delete and recreate — on AlreadyExists, delete the existing provider and create a fresh one with current credentials
- Hybrid — try create; on AlreadyExists, update or delete+recreate
The right choice depends on whether credentials could change between runs. Option 1 is simpler but may leave stale credentials; option 2 is safer for credential rotation.
Related
There is a related idempotency gap in InferenceLayer.Install() (inference.go:56-69) where the idempotency guard checks whether secrets exist but not whether the values are correct, allowing stale or incorrect values to persist across re-runs.
Reported by @mrizzi
Problem
When running
fullsend runtwice against the same gateway (e.g. during local testing), the second run fails at the "Ensuring provider" step with:EnsureProviderininternal/sandbox/sandbox.gocallsopenshell provider createand treats any error as a hard failure. There is no handling for theAlreadyExistscase — no silent success, no delete-and-recreate fallback, and no update path. The function comment says "creates or updates a provider" but only the create path is implemented.The current workaround is to manually delete providers between runs (
openshell provider delete github jira), which makes iterative local testing painful. Any failed or interrupted run leaves stale providers that block the next run.Expected Behavior
EnsureProvidershould be idempotent, consistent with the codebase's conventions:Providerinterface requiresProvision()implementations to be idempotentEnsureGateway()is documented as idempotent in ADR 0030Possible Approaches
The right choice depends on whether credentials could change between runs. Option 1 is simpler but may leave stale credentials; option 2 is safer for credential rotation.
Related
There is a related idempotency gap in
InferenceLayer.Install()(inference.go:56-69) where the idempotency guard checks whether secrets exist but not whether the values are correct, allowing stale or incorrect values to persist across re-runs.Reported by @mrizzi