Parallelise and throttle the id-minter with a generalised find_work state machine

[kenoir]

Context

During the #6445 reindex, 2.6M records landed in three consecutive id-minter windows. All three executions failed (one Lambda timeout, two OOMs), and window mode has no backfill, so the records were silently skipped until manually replayed. wellcomecollection/catalogue-pipeline#3524 added a manual replay lever and wellcomecollection/catalogue-pipeline#3527 fixed lookup throughput (roughly 16x), but the structure is unchanged: one Lambda processes a whole window serially, the 15-minute schedule cadence and the 900s Lambda timeout are essentially the same number so there is no headroom band, and a failed window is still skipped silently. #6328 framed the same options before the Python port.

Approach

Reuse the image inferrer's find_work pattern (wellcomecollection/catalogue-pipeline#3413): a small discovery Lambda scans the source index for ids in the window, partitions them to S3 (pass-by-reference, keeping the Step Functions payload under the 256 KB limit), and a Map state fans out workers with bounded concurrency. Rather than copying it, generalise it so both consumers share one implementation:

• Python: extract a generic find-and-partition core; the inferrer and id-minter keep thin per-service entrypoints, built as separate Lambdas from the unified image so IAM and ES credentials stay scoped per service.
• Terraform: a shared state machine module owning the ConstructEvent input guards, the find-work Lambda, the Map skeleton with MaxConcurrency, post-Map failure aggregation, alarms and the schedule. The per-service worker block (ECS runTask for the inferrer, lambda:invoke for the id-minter) is injected by the caller, since the two share almost nothing.

For the id-minter specifically: MaxConcurrency is pinned to the RDS budget (the measured sweet spot during the #6445 recovery was 6), partitions are sized by work count (~10k works keeps worst-case identifier density well inside the Lambda timeout at post-#3527 throughput), and failed partitions fail the execution rather than being tolerated, because unlike the inferrer no later window re-covers a missed indexed_at range. Concurrent minting is already safe: FOR UPDATE SKIP LOCKED plus the FOR SHARE race detection, exercised 6-wide during the recovery.

Phases

1. Extract the shared module (Extract a shared find_work state machine module from the image inferrer catalogue-pipeline#3536). Move the inferrer state machine into the new module, behaviour-identical except two deliberate diffs: the Unpacker doesn't have access to uploads in the ingests bucket from Archivematica #3524-style input guards (giving the inferrer explicit-window replays, currently only possible with a synthetic scheduled_time) and a post-Map state that aggregates partition failures (today the partition_failed flags are recorded but nothing reads or alarms on them). Apply gate: the inferrer plan shows a definition-only change matching those two diffs.
2. Fan out the id-minter (Fan out the id-minter over partitions via the shared find_work module catalogue-pipeline#3537, stacked on tidy data-science terraform #3536). New id_minter find_work step and Lambda (plus a CI deploy-matrix entry), instantiate the module in place of the current two-state machine, resolve S3 partition refs in the id-minter Lambda, and derive per-partition job_ids (fixing the minute-granular collision on report names). Exercise on 2026-07-03 with an explicit-window replay over a known dense Full reindex of data into the new pipeline #6445-era window before relying on the schedule.

Both phases need a manual terraform apply per pipeline date.