TEI delete messages silently dropped due to SQS visibility timeout shorter than internal processing delay — bured6mf never suppressed
Summary
bured6mf (source identifier MSAmer9) remained live in the catalogue indefinitely after its TEI manuscript file was renamed (Spanish/WMS_Amer.9.xml → Spanish/MS_Amer_9.xml, xml:id MSAmer9 → MS_Amer_9, now correctly represented as bb7j6vqh). The deletion signal for the old identifier was generated correctly upstream, but was silently and repeatedly dropped by both tei_id_extractor and tei_adapter, because each service's SQS queue had a visibility timeout shorter than the deliberate in-process delay it applies to delete messages.
Root cause
Both services intentionally delay processing of *Deleted messages, to avoid misinterpreting a file rename (delete-of-old + add-of-new) as a genuine deletion before the corresponding "changed" message has had a chance to land:
- tei_id_extractor: TeiIdExtractorWorkerService.processDeleted → .delay(config.deleteMessageDelay), 30 minutes.
- tei_adapter: TeiAdapterWorkerService.delayDeleted → .delay(delay), 2 minutes.
Both messages sit in an in-memory Pekko stream buffer during this delay, without ever being acknowledged on the SQS queue. If the queue's visibility timeout is shorter than the delay, SQS makes the message visible again for redelivery before the delay has even finished once. The redelivered copy restarts its own delay, gets interrupted again, and so on — receive count climbing every cycle — until maxReceiveCount is exceeded and the message lands in the DLQ. Critically, nothing ever throws during this loop, so there is no exception for any logging to catch — this is why the failure was completely invisible in application logs.
| Service |
Delete delay |
Visibility timeout (broken, original) |
Visibility timeout (manual console fix, tested) |
Visibility timeout (final, Terraform formula) |
tei_id_extractor |
30 min (1800s) |
630s (10.5 min) — rds_lock_timeout_seconds + 30 |
2400s (40 min) |
2400s (40 min) — delete_delay + 600s buffer |
tei_adapter |
2 min (120s) |
60s |
300s (5 min) |
720s (12 min) — delete_delay + 600s buffer |
Evidence
- bured6mf's source identifier (MSAmer9) and bb7j6vqh's (MS_Amer_9) confirmed via Elasticsearch + the id_minter's SQL ground truth to be the same manuscript.
- The old file path was confirmed genuinely absent from tei_updater's current tree snapshot, and tei_id_extractor's RDS PathIdTable row for MSAmer9 still pointed at the stale path — confirming the delete was generated but never completed processing.
- tei-adapter-store DynamoDB item for MSAmer9 remained TeiChangedMetadata (never TeiDeletedMetadata) throughout, despite multiple delete-message replay attempts.
- Fix confirmed empirically: after raising both queues' visibility timeouts, republishing {"id":"MSAmer9","timeDeleted":"2026-08-04T13:12:00Z","type":"TeiIdDeletedMessage"} directly to tei-id-extractor-output resulted in the message surviving its full 2-minute delay cleanly in-flight (no redelivery loop), and tei-adapter-store's DynamoDB item for MSAmer9 correctly flipped from TeiChangedMetadata (v5) to TeiDeletedMetadata (v6).
Fix
Applied manually in the AWS console first (to unblock and verify)
- tei_adapter/terraform/tei_adapter.tf: queue_visibility_timeout_seconds 60 → 300 (5 min)
- tei_adapter/terraform/tei_id_extractor.tf: queue_visibility_timeout_seconds 630 → 2400 (40 min)
Impact
This would have silently affected every TeiIdDeletedMessage/TeiPathDeletedMessage processed by these services since the visibility timeout values were set — not unique to MSAmer9. Worth an audit for other orphaned/duplicate TEI records that may share this same root cause?
TEI delete messages silently dropped due to SQS visibility timeout shorter than internal processing delay — bured6mf never suppressed
Summary
bured6mf (source identifier MSAmer9) remained live in the catalogue indefinitely after its TEI manuscript file was renamed (Spanish/WMS_Amer.9.xml → Spanish/MS_Amer_9.xml, xml:id MSAmer9 → MS_Amer_9, now correctly represented as bb7j6vqh). The deletion signal for the old identifier was generated correctly upstream, but was silently and repeatedly dropped by both tei_id_extractor and tei_adapter, because each service's SQS queue had a visibility timeout shorter than the deliberate in-process delay it applies to delete messages.
Root cause
Both services intentionally delay processing of *Deleted messages, to avoid misinterpreting a file rename (delete-of-old + add-of-new) as a genuine deletion before the corresponding "changed" message has had a chance to land:
Both messages sit in an in-memory Pekko stream buffer during this delay, without ever being acknowledged on the SQS queue. If the queue's visibility timeout is shorter than the delay, SQS makes the message visible again for redelivery before the delay has even finished once. The redelivered copy restarts its own delay, gets interrupted again, and so on — receive count climbing every cycle — until maxReceiveCount is exceeded and the message lands in the DLQ. Critically, nothing ever throws during this loop, so there is no exception for any logging to catch — this is why the failure was completely invisible in application logs.
tei_id_extractorrds_lock_timeout_seconds + 30delete_delay + 600s buffertei_adapterdelete_delay + 600s bufferEvidence
Fix
Applied manually in the AWS console first (to unblock and verify)
Impact
This would have silently affected every TeiIdDeletedMessage/TeiPathDeletedMessage processed by these services since the visibility timeout values were set — not unique to MSAmer9. Worth an audit for other orphaned/duplicate TEI records that may share this same root cause?