Design: forget stale accounts — deprovisioning lifecycle for onboarded BYOA accounts

[scttfrdmn]

Problem

Onboarding an account is one-way. There is no path by which spore.host ever concludes "this account is gone" — so the state it leaves behind is permanent, and one piece of it is a genuine security hazard.

Three places encode this, all by construction rather than oversight:

• The account registry is append-only. spore-host/lambda/portal-phone-home/registry.go exposes PutAccount and GetAccount and nothing else — no delete, no status, no last-seen. A registration cannot be retired.
• The reaper's account list is static config. lambda/ttl-reaper/main.go:985 derives each sweep subdomain from r.accounts, populated purely from REAPER_ROLE_ARNS. An account is swept because a human typed it into a deploy, and stops being swept because a human edited it out.
• dns-updater persists nothing. It writes Route53 and forgets. Route53 is therefore the only durable record that an account ever existed.

Two distinct failure modes

A. Forgotten-but-still-listed. The account goes away; its ARN stays in REAPER_ROLE_ARNS. The reaper burns a failing assume-role every 10 minutes forever, and each failure increments sum.Errors (main.go:416,439) — the field that is supposed to mean investigate this. Permanent expected failures train the operator to ignore the error count.

B. Forgotten-and-never-listed — silent, and live today. Records exist under {base36}.spore.host for an account absent from REAPER_ROLE_ARNS. Nothing sweeps it, nothing errors, nothing reports it.

Concrete instance: win11-test.4zlw3a1t.spore.host → 13.223.189.34. Subdomain 4zlw3a1t decodes to account 390967728545, which is not in REAPER_ROLE_ARNS, has no cross-account role, and appears nowhere in any spore-host repo. The IP is back in the general us-east-1 EC2 pool and no longer responds — so this name will eventually resolve to an unrelated instance once AWS reassigns the address. Stale DNS pointing at a stranger, not merely a dead name. (Found while verifying #438; deliberately left in place.)

What can and cannot be expired

The thing you would most want a TTL on is not ours to expire. deployment/cloudformation/ttl-reaper-cross-account-role.yaml creates spawn-ttl-reaper-ec2 in the customer's account; spore.host holds only the trust relationship. We cannot delete it — only they can. An IAM role with no caller is also inert and free, so no cost pressure forces the issue.

What is ours, ranked by hazard:

Artifact	Owner	Hazard	Wants
Route53 records under {base36}.{domain}	spore.host	Real — IP returns to the EC2 pool; name resolves to a stranger	Shortest grace period, not the longest
Registry row (spore-portal-accounts)	spore.host	None (costs nothing; it is the audit trail)	status + lastSeenAt, not deletion
REAPER_ROLE_ARNS entry	spore.host (config)	Noise — poisons Errors	Auto-quiesce after sustained failure
spawn-ttl-reaper-ec2 role	customer	None	Nothing — out of our control

Note the DNS row inverts the usual instinct: the security consequence argues for expiring records sooner, not keeping them around just in case.

The liveness signal already exists, for free

The reaper assumes every role every 10 minutes. That is already a paid-for liveness probe, and it distinguishes exactly the states that matter:

• assume succeeds, zero managed instances for N days → dormant but reachable
• assume fails, role not found → the customer uninstalled

Role deletion is the natural uninstall gesture. No new API and no customer action beyond what they would already do.

Two traps that make the naive version dangerous

1. Correlated failure is indistinguishable from mass uninstall. The reaper's role ARN is …spawn-ttl-reaper-production-TTLReaperFunctionRole-ZJ84YZ2dCPei — a CloudFormation-generated physical ID. Recreating that stack changes the suffix and breaks every customer's trust policy simultaneously. A TTL that deprovisions on assume-role failure would then forget the entire customer base because we redeployed. Never act on a single failure; explicitly refuse to act when all accounts fail at once. Same instinct as the sweep's existing "never delete against a partial live set" guard (main.go:987-992).

2. We lose the ability to verify at the exact moment we would act. "Only the role is left, no other resources" requires a working assume-role to confirm — DescribeInstances is how we know the account is empty. Once the role is gone that check is impossible, so we would be deciding blind. win11-test is precisely this case: records present, emptiness unverifiable except by the weak inference that the IP stopped responding.

Proposed policy

Grace, then narrow, then never silently delete what cannot be verified.

• Unreachable — assume-role fails for K consecutive runs while other accounts succeed: mark the registry row unreachable, stop counting it into Errors. No deletion.
• Dormant — assume-role works and zero managed instances for N days: safe to expire that account's DNS records, because emptiness is provable.
• Unmanaged subdomain — records under a {base36} we hold no credentials for: report, never delete. We cannot distinguish "abandoned" from "someone forgot to add this account to REAPER_ROLE_ARNS," and deleting in the second case breaks a working spore.
• Explicit offboard — an operator/customer-initiated deprovision that may delete records, because intent is stated rather than inferred.

K and N are the open decisions. Suggest K=6 (one hour) and N=30 days, both configurable.

Suggested split

Two separable pieces; the first is worth doing regardless of how the policy lands.

• Orphan-subdomain report in the reaper. (shipped in feat(ttl-reaper): report account subdomains with no credentials (#457) #458) One ListResourceRecordSets over the zone → distinct {base36} labels → decode (strconv.ParseUint(label, 36, 64), the inverse of dns.EncodeAccountID) → diff against the configured account set → log + count the unmanaged ones. Report-only, no deletion. Small, safe, and unit-testable without AWS in the same style as sweepableRecord. Would have surfaced 4zlw3a1t the day it appeared.
• Registry lifecycle fields (shipped in feat(portal-phone-home): account lifecycle state machine (spawn#457) spore-host#490; cross-repo wiring to the reaper still open — see the comments) (status, lastSeenAt) + the state machine above. Touches the onboarding contract, so it deserves its own decision.

Caveat on liveness tests

Checking AWS Organizations membership is not a general staleness test. BYOA accounts belong to customers and are legitimately outside our org. AccountNotFoundException only proved staleness for 4zlw3a1t because that happened to be one of our own test accounts.

---

Filed from the #438 verification work; see #438 for the sweep's current safety contract.

[scttfrdmn]

Checkbox 2 (registry lifecycle + the state machine) is implemented in spore-host/spore-host#490 — all CI green. K=6 / N=30d as suggested here, both configurable on LifecyclePolicy.

Landed in lambda/portal-phone-home/lifecycle.go (the schema's owner) rather than in the reaper, so the fields and the transitions live next to the table they describe.

Both traps from this issue are enforced and mutation-verified:

• Correlated failure — ApplyProbes changes nothing when every probe fails. Neutering that guard fails two tests. The sharp edge turned out to be the single-account deployment, where "all probes failed" and "this one account failed" are literally the same observation; the guard resolves it as uninformative, which is the right answer but worth naming since it means a one-account deploy can never auto-transition on failure at all.
• Unverifiable at the moment of action — DNSExpiryEligible() returns true for only dormant (emptiness proven through a working DescribeInstances) and offboarded (stated intent). unreachable deletes nothing. Unknown/future status strings return false, so a typo can't authorize a deletion.

Two things worth flagging beyond the plan:

1. Mutation testing caught a real bug — the unreachable branch guarded against re-entering its own state but not against overwriting offboarded. That would have downgraded a stated human decision to a weaker inferred one and stripped the DNS-expiry eligibility offboarding deliberately granted. Fixed, with the reasoning in the code.

2. lambda/portal-phone-home was absent from the CI test matrix entirely — so its tests had never run, including TestValidate_SecurityInvariant, which covers the SigV4 verified-principal check that is the whole authorization model for BYOA onboarding. Added in the same PR at a 70% floor (actual 76.8%). Unrelated to this issue, but found on the way through.

Schema compatibility: every new field is omitempty and AccountStatus() maps "" → active, so existing rows need no backfill.

Still open — the cross-repo wiring, which is why I'm leaving this issue open: ApplyProbes has no production caller. The reaper needs to (a) collect a ProbeResult per account per run, (b) get dynamodb:Scan + dynamodb:UpdateItem on spore-portal-accounts in its own execution role, and (c) consult DNSExpiryEligible before expiring records. I deliberately did not add those IAM grants to the phone-home role in #490, since they'd be permissions with no caller.

One design question that wiring will force, and I'd rather decide it deliberately: the reaper currently scans accounts from REAPER_ROLE_ARNS config, while the registry is the durable list. Once the reaper reads the table, there are two sources of truth for "which accounts exist". The clean end state is config seeding the registry and the registry driving the sweep — but that changes how an operator adds an account, so it isn't a silent refactor.

[scttfrdmn]

Filed the wiring as spore-host/spore-host#491, and it lands somewhere different from what this issue proposed — worth recording why here, since the design above says the reaper should do the probing.

The reaper cannot assume the registry's roles. There are two distinct cross-account roles, and onboarding only ever creates one:

	spawn-ttl-reaper-ec2	spore-portal-onboard
Created by	manual CFN deploy per account	portal quick-create / spawn onboard
Trusts	the reaper Lambda role	the phone-home Lambda role
ExternalId	static spawn-ttl-reaper	per-account, high-entropy, in the registry
Listed in	REAPER_ROLE_ARNS	spore-portal-accounts
Permissions	Describe + Terminate + FSx	Describe + RunInstances, PassRole, SSM

So every roleArn the reaper would read from the registry is one its own trust policy excludes.

The failure mode is the interesting part. Config-derived accounts would have kept succeeding while every registry-derived account failed — and the correlated-failure guard only trips when all probes fail, so it would not have fired. Within K=6 runs (one hour), every portal-onboarded account would have been marked unreachable.

The guard cannot catch this by construction: from the inside, "a set of accounts whose role we are not trusted by" is indistinguishable from "a set of customers who uninstalled". That is precisely trap 1 from this issue, arriving through a door the guard does not cover. Good argument for the guard being necessary but not sufficient, and for keeping the probe's trust boundary explicit rather than inherited.

Where it goes instead: a scheduled Lambda in the infra account that assumes the portal onboarding role. The trust already exists (PortalPhoneHomeLambdaRole is the trusted principal), the registry already holds the matching ExternalId, and spore-portal-onboard already grants ec2:DescribeInstances — which is exactly what proving emptiness requires. Works for every already-registered account with no template change and no re-onboarding, and needs only Describe, never Terminate.

And the two-sources-of-truth question I raised earlier dissolves rather than needing a decision. REAPER_ROLE_ARNS is the TTL-enforcement list (who can we terminate in); the registry is the lifecycle list (who onboarded, are they still there). Different roles, different trust, different questions. Treating them as competing was a framing error on my part — nothing needs reconciling.

The reaper does still get one small change, tracked in #491: consult DNSExpiryEligible before expiring a subdomain's records. That is read-only access to a table in its own account — no assume-role, no trust problem.

Rejected on the way: adding the reaper's role to the onboarding template. It requires re-onboarding every existing account (a template change doesn't alter deployed stacks) and puts a Terminate-capable role on the portal's onboarding path, collapsing the current split between the portal's launch/manage role and the reaper's out-of-band kill role.