feat(oabctl): programmatic delete API with explicit target contract

docs/oabctl.md

@@ -219,6 +219,11 @@ spec:
       securityGroups: [sg-xxx]
 ```
 
+`metadata.namespace` must be hyphen-free (`[a-z0-9]+`); `metadata.name` may
+contain `-`. Physical resources are named `oab-{namespace}-{name}`, and a
+hyphen-free namespace keeps that identity unambiguous (see the teardown note
+below for the legacy-namespace policy).
+
 ### Ingress — inbound webhooks (Telegram / LINE)
 
 Discord bots are outbound-only and need no ingress. Webhook platforms (Telegram,
@@ -365,18 +370,70 @@ must still be registered manually in the LINE Developers console.
 > reminded, not verified).
 >
 > **Teardown:** `oabctl delete oabservice <name>` (or `oabctl delete -f <manifest>`,
-> using the same file `apply -f` deployed it from) permanently removes the bot's
-> per-bot ingress resources — its exact Cloud Map service (resolved by the ECS
-> service's own registry ARN, not a name search, so same-named bots in different
-> VPCs/environments can't collide) and its HTTP API (`oab-webhook-<ns>-<name>`,
-> including the API resource itself this time, since the bot is gone for good) —
-> on a best-effort basis (it never blocks service deletion). If you instead edit
-> a manifest to remove `spec.ingress` while keeping the bot, `apply` runs the same
-> Cloud Map + routes/integration/stage cleanup automatically, but **keeps the HTTP
-> API** in case ingress is re-added later. The **shared** VPC Link and the
-> security-group inbound rule are always left in place for other bots. If the
-> Cloud Map service still has registered instances, teardown retries for ~25s
-> before falling back to a warning with the manual cleanup command.
+> using the same file `apply -f` deployed it from) first captures the exact ECS
+> cluster/service ARNs, ECS registry ARN, matching HTTP API ID, configured
+> control-plane bucket, caller partition/account/region, and ECS service
+> incarnation (`createdAt`) in `delete-checkpoints/<namespace>/<name>.json`. The
+> checkpoint is written before ECS mutation and removed only after all
+> dependent and S3 cleanup succeeds, so rerunning the same command safely
+> resumes a partial delete. ECS HTTP-200 failures, ambiguous drain responses,
+> empty service responses, and a missing service fail closed for strict/programmatic
+> cleanup; a matching checkpoint authorizes retry only when ECS explicitly reports
+> exactly one `MISSING` failure with zero services, or an `INACTIVE` response from
+> the original service incarnation. The legacy CLI warns on a drain timeout,
+> continues dependent/S3 cleanup, retains the checkpoint for retry, and continues
+> after individual S3 object failures. The control-plane bucket must enforce default
+> server-side encryption and S3 versioning for durable checkpoint recovery;
+> oabctl writes application JSON but does not weaken or validate those
+> bucket-level policies per request. The programmatic `delete_services` API
+> keeps exact-identity and checkpoint cleanup failures fatal for safe retry.
+>
+> API cleanup never selects the first same-named API: duplicate names fail
+> closed, and the sole candidate is checkpointed only when its integration URI
+> equals the ECS registry ARN. Cloud Map is deleted only by the service ID
+> parsed from a structurally and boundary-validated Cloud Map service ARN.
+> Exact-ID NotFound is idempotent; other cleanup errors retain the checkpoint.
+> There is no name-only API, Cloud Map, or S3 orphan cleanup path. Apply and
+> delete share one injective identity rule: namespaces must match `[a-z0-9]+`
+> and names must match `[a-z0-9][a-z0-9-]*`; this rejects `/`, whitespace,
+> `_`, `.`, and Unicode before any ECS, API, or S3 request. Namespaces must not
+> contain `-` on apply and programmatic delete (names may contain it), so the
+> physical `oab-{namespace}-{name}` identity always maps back to exactly one
+> `namespace`/`name` pair. The legacy CLI delete path permits hyphenated
+> namespaces only when the remaining S3-safe character rules pass. Manifest
+> validation and programmatic delete reject hyphenated namespaces before any
+> AWS call, and before mutating anything both apply and delete verify in the
+> control plane that no other recorded logical pair (for example legacy
+> `prod-team/bot` versus `prod/team-bot`) claims the same physical name —
+> contested identities fail closed. Legacy deployments created under a
+> hyphenated namespace cannot be re-applied, but `oabctl delete` still accepts
+> them (with a warning) under that same ownership check; migrate by deleting
+> and re-creating them under a hyphen-free namespace. If a per-target
+> `ingress-teardown-checkpoints/<namespace>/<name>.json` record exists, delete
+> fails before destructive mutation so it cannot discard unfinished exact
+> cleanup identity. Re-run the ingress-free apply to completion, then retry
+> delete.
+>
+> Programmatic apply and delete do not provide an internal same-target lock.
+> Callers must serialize mutations for the same AWS account, Region,
+> control-plane bucket, ECS cluster, and physical service identity. This
+> includes every alias-equivalent logical pair that could map to the same
+> `oab-{namespace}-{name}` value (for example `prod/team-bot` and
+> `prod-team/bot`). After an accidental overlap, stop concurrent writers,
+> inspect retained checkpoints, then explicitly re-apply the desired state or
+> retry delete.
+>
+> If you edit a manifest to remove `spec.ingress` while keeping the bot, `apply` first
+> stores every exact ECS registry ARN only when the previously stored OAB
+> manifest owned ingress (or resumes an already-valid checkpoint); an arbitrary
+> attached registry is never enough. The resulting exact-identity record is
+> stored at `ingress-teardown-checkpoints/<namespace>/<name>.json`; it clears only
+> API wiring bound to those ARNs, detaches all registries from ECS, waits until
+> the detach is observable, and only then deletes each exact Cloud Map service.
+> It keeps the HTTP API resource so its URL can survive re-enabling ingress and
+> removes the checkpoint only after the full apply succeeds, so retries cannot lose
+> cleanup identity. The **shared** VPC Link and security-group inbound rule are
+> always left in place for other bots.
 >
 > **Changing `paths`:** `apply` prunes routes on the bot's API that are no longer
 > in the manifest's `ingress.paths`, so renaming or removing a webhook path never
@@ -419,6 +476,9 @@ spec:
 ### Design Principles
 
 - **Manifest = infra desired state** — image, CPU, networking
+- **Apply failure recovery** — apply persists the manifest before ECS mutation;
+  if a later AWS/configuration step fails, correct the cause and re-run apply to
+  reconcile that persisted desired state. Apply has no rollback checkpoint.
 - **Agent config is external** — `configFrom` points to config.toml (managed via `--sync`)
 - **Secrets resolved by OpenAB** — `[secrets.refs]` in config.toml, not in manifest
 - **Runtime-agnostic spec** — same top-level fields regardless of ECS or K8S

operator/README.md

@@ -75,16 +75,19 @@ commands reference.
 
 ## Library API
 
-The crate exposes a deliberately narrow manifest + apply facade for control
-planes that should not shell out to the CLI:
+The crate exposes a deliberately narrow manifest + apply/delete facade for
+control planes that should not shell out to the CLI:
 
 ```rust,no_run
-use oabctl::{apply_manifests, ApplyOptions, OABServiceManifest};
+use oabctl::{
+    apply_manifests, delete_services, ApplyOptions, DeleteOptions, DeleteTarget,
+    OABServiceManifest,
+};
 
-async fn deploy(
+async fn reconcile(
     aws: &aws_config::SdkConfig,
     manifest: OABServiceManifest,
-) -> Result<(), oabctl::ApplyError> {
+) -> Result<(), Box<dyn std::error::Error>> {
     let report = apply_manifests(
         aws,
         &[manifest],
@@ -94,20 +97,90 @@ async fn deploy(
     for service in report.services {
         println!("{}: {:?}", service.ecs_service_name, service.action);
     }
+
+    delete_services(
+        aws,
+        &[DeleteTarget::new("prod", "bot")],
+        // Omit the bucket override to use OAB_CONTROL_PLANE_BUCKET,
+        // then oab-control-plane-{account} from the caller identity.
+        &DeleteOptions::new("production-cluster"),
+    )
+    .await?;
     Ok(())
 }
 ```
 
-Programmatic apply emits no progress to process-global stdout/stderr. Success
-returns per-service actions, webhook URLs, and warnings; reconciliation errors
-identify the failed service and include the report completed before the failure.
-Both CLI and programmatic apply verify that the target cluster exists and is
-`ACTIVE` before mutation, so the caller identity requires
-`ecs:DescribeClusters`. This ECS action does not support resource-level
+Programmatic apply/delete emit no progress to process-global stdout/stderr.
+Apply success returns per-service actions, webhook URLs, and warnings;
+reconciliation errors identify the failed service and include the report
+completed before the failure. Both CLI and programmatic apply verify that the
+target cluster exists and is `ACTIVE` before mutation, so the caller identity
+requires `ecs:DescribeClusters`. This ECS action does not support resource-level
 permissions; its IAM statement must use `Resource: "*"`.
-The library never reads `~/.oabctl/config.toml`: set
-`with_control_plane_bucket(...)` explicitly when needed, otherwise bucket
-resolution uses `OAB_CONTROL_PLANE_BUCKET` and then the caller's AWS account.
+
+The library never reads `~/.oabctl/config.toml`. Apply and delete resolve the
+control-plane bucket through the shared chain: an explicit
+`with_control_plane_bucket(...)` override, then `OAB_CONTROL_PLANE_BUCKET`, then
+`oab-control-plane-{account}` from the caller identity. Before deleting ECS,
+`delete_services` stores the resolved bucket, caller partition/account/region,
+canonical ECS cluster/service ARNs, the service `createdAt` incarnation, and
+exact ingress IDs in `delete-checkpoints/<namespace>/<name>.json`. The checkpoint
+is written before ECS mutation and removed last. Initial DescribeServices
+responses must identify exactly one expected live service; missing, empty-success,
+ambiguous, or mixed-failure responses fail closed. Retries use only checkpointed
+IDs after ECS explicitly reports exactly one `MISSING` failure with zero services
+or a matching original `INACTIVE` incarnation. Duplicate APIs fail closed,
+and a sole same-name API is checkpointed only when an integration URI exactly
+matches the ECS registry ARN.
+There is no name-only API, Cloud Map, or S3 orphan cleanup fallback. If a
+per-target apply-side `ingress-teardown-checkpoints/<namespace>/<name>.json`
+record exists, delete fails before destructive mutation so it cannot discard
+unfinished exact cleanup identity. Re-run the ingress-free apply to completion,
+then retry delete.
+
+The manifest is persisted as the infrastructure desired state before ECS
+mutation. If a later AWS/configuration step fails, correct the underlying issue
+and re-run `apply`/`apply_manifests` to reconcile the persisted desired state;
+apply has no rollback checkpoint.
+The control-plane bucket must enforce default server-side encryption and S3
+versioning for durable checkpoint recovery. The library writes application JSON
+but does not weaken or validate those bucket-level policies per request. The
+legacy CLI warns on a drain timeout, continues dependent/S3 cleanup, retains the
+checkpoint for retry, and continues after individual S3 object failures, while
+`delete_services` keeps exact-identity and checkpoint cleanup failures fatal so
+callers can retry safely.
+
+Apply and delete share one injective logical-identity rule: namespaces must not
+contain `-`; names may contain it, so the physical `oab-{namespace}-{name}`
+identity always parses back to exactly one `namespace`/`name` pair. Namespaces
+must match `[a-z0-9]+` and names must match `[a-z0-9][a-z0-9-]*`; this also
+prevents `/`, whitespace, `_`, `.`, and Unicode from becoming S3 path
+components. Manifest validation (CLI apply, fleet expansion, and
+`apply_manifests`) and `delete_services` all reject hyphenated namespaces before
+any AWS call. The legacy CLI delete path permits hyphenated namespaces only
+when the same S3-safe character rules otherwise pass. Before mutating, apply
+and every delete entry point additionally verify in the control plane that no
+*other* recorded logical pair (manifest, delete checkpoint, or
+ingress-teardown checkpoint) claims the same physical name, and fail closed
+with the colliding pair named in the error.
+
+Legacy policy: deployments created under a hyphenated namespace before this
+rule cannot be re-applied — apply fails with a migration message. They remain
+deletable through `oabctl delete`, which accepts a hyphenated namespace for
+teardown, prints a warning, and relies on the same control-plane ownership
+check to refuse contested physical names. Migrate by deleting the legacy
+deployment and re-creating it under a hyphen-free namespace.
+
+Programmatic apply and delete do not provide an internal same-target lock.
+Callers must serialize mutations for the same AWS account, Region, control-plane
+bucket, ECS cluster, and physical service identity. This includes every
+alias-equivalent logical pair that could map to the same
+`oab-{namespace}-{name}` value (for example `prod/team-bot` and
+`prod-team/bot`). The complete delete target set is validated before any AWS
+request; duplicate logical targets are rejected. If overlapping calls occur,
+stop concurrent writers, inspect the retained checkpoint, then explicitly
+re-apply the desired state or retry delete.
+
 For `aws-sm://<secret-id>#<json-key>`, a non-ARN `<secret-id>` requires the
 caller to have `secretsmanager:DescribeSecret`; full-ARN shorthand does not
 need that lookup.

operator/schema/oabservice-v2.json

@@ -35,7 +35,7 @@
       "required": ["name", "namespace"],
       "properties": {
         "name": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]*$" },
-        "namespace": { "type": "string", "pattern": "^[a-z0-9][a-z0-9-]*$" },
+        "namespace": { "type": "string", "pattern": "^[a-z0-9]+$" },
         "generation": { "type": "integer", "minimum": 0 }
       },
       "additionalProperties": false
@@ -45,7 +45,7 @@
       "required": ["name", "namespace"],
       "properties": {
         "name": { "type": "string" },
-        "namespace": { "type": "string" }
+        "namespace": { "type": "string", "pattern": "^[a-z0-9]+$" }
       },
       "additionalProperties": false
     },