Guard against spored/reaper IAM drift: code calls AWS APIs the role doesn't grant (recurring class)

[scttfrdmn]

Pattern

The same class of bug has now bitten three times: in-instance/out-of-band code (spored, ttl-reaper) calls an AWS API that the relevant IAM role/policy does not grant, so the call fails with AccessDenied — and because these paths are best-effort and under-logged, it manifests as a silent no-op, not a loud failure.

Confirmed instances:

• fix(reaper): grant cross-account role fsx:DescribeFileSystems + DeleteFileSystem (#210) #212 — ttl-reaper's FSx-cleanup path called fsx:* but the cross-account role granted EC2 only (silent per-region AccessDenied, masked by dry-run).
• spored did not mount the async-created ephemeral FSx after it reached AVAILABLE (no mount, no log lines) — #194 not firing #221 — spored's async-FSx-mount called fsx:DescribeFileSystems / fsx:CreateDataRepositoryAssociation but the spored inline policy had no fsx:* (silent no-op; also a startup race).
• [HIGH] dns-updater accepts unsigned identity docs + default-allows cross-account → Route53 spoofing #173 (adjacent) — spored's DNS path needed lambda:InvokeFunctionUrl on the dns-updater fn (added in the cutover).

Why it keeps happening

The IAM policy (pkg/aws/iam.go buildInlinePolicy for spored; lambda/ttl-reaper/template.yaml for the reaper) is maintained by hand, separately from the code that makes the calls. Adding an SDK call doesn't fail any build/test if the matching grant is missing — it only fails at runtime, in an account, often silently.

Ideas (pick one or more)

1. A test that greps the agent/reaper packages for client.<Service> SDK call sites and asserts each service appears in the inline policy (a coarse but cheap drift guard — would have caught all three).
2. Centralize the spored required-action list in one place that both buildInlinePolicy and a doc/test consume, so adding a call forces a policy edit in the same change.
3. Make these best-effort AWS paths log AccessDenied loudly (distinct from "nothing to do") so a missing grant is diagnosable from logs without a live repro — directly addresses the "silent no-op" half. (spored did not mount the async-created ephemeral FSx after it reached AVAILABLE (no mount, no log lines) — #194 not firing #221 Q3 asked for exactly this on the FSx path.)
4. A periodic spawn doctor-style IAM self-check that dry-run/simulates the spored role against its required actions.

Low-glamour, high-leverage: each occurrence has cost a live debugging session. Filing so the next FSx/DNS/whatever feature doesn't rediscover it.