Bound search concurrency, S3 reads, and attachment resource usage

[jzhao234]

Problem

Each exact/fuzzy search currently starts a new daemon thread and creates a new S3 client. Searches have an individual runtime limit, but there is no global or per-user concurrency bound. Repeated requests can therefore create many simultaneous S3 scans and consume CPU, memory, threads, network bandwidth, and S3 request budget.

Full messages and attachment payloads are also read into memory and written to temporary disk without explicit application-level object or attachment size limits.

Desired behavior

Add bounded resource controls for searches, S3 reads, parsing, and temporary attachment storage.

At minimum:

• Replace unbounded thread creation with a bounded executor, queue, or semaphore.
• Limit concurrent searches globally and per authenticated user/session.
• Reject or coalesce duplicate active searches from the same user.
• Bound query/filter/keyword lengths and the number of search terms/date windows.
• Set explicit S3 connect/read timeouts and a bounded retry policy.
• Check S3 object size before downloading and enforce maximum raw-message and attachment sizes.
• Bound temporary attachment disk usage and always remove partial files after errors/cancellation.
• Preserve cancellation and per-job runtime limits while queued or executing.
• Return clear 429 or 413 responses for concurrency and size limits rather than failing unpredictably.

Acceptance criteria

• Starting more than the configured per-user/global search limit does not create additional worker threads or S3 scans.
• Excess work receives a deterministic bounded response (429 or an explicitly documented queued state).
• Oversized messages and attachments are rejected with 413 before unbounded memory/disk consumption.
• Query strings, individual filters, keyword counts, and date-window counts have documented upper bounds enforced server-side.
• S3 operations use explicit timeouts and bounded retries.
• Cancellation works for queued and running jobs.
• Temporary and partial attachment files are removed after success, failure, timeout, and cancellation.
• Tests exercise concurrency saturation, oversized S3 objects, slow/failing S3 calls, cancellation, and cleanup without requiring a live S3 bucket.
• Relevant limit hits are observable through non-sensitive logs or metrics.

Security/operations impact

These limits protect availability and control S3 costs if an authenticated browser is automated, a user repeatedly starts expensive searches, or the archive contains unexpectedly large/malformed messages.