Skip to content

feat: Hardened FileSystemTransport (v1.1.0) - #1

Merged
wieslawgolec merged 6 commits into
mautic7.xfrom
feature/hardened-filesystem-transport
Oct 5, 2026
Merged

wieslawgolec merged 6 commits into
mautic7.xfrom
feature/hardened-filesystem-transport

Conversation

@wieslawgolec

Copy link
Copy Markdown
Owner

Summary

Hardened filesystem queue transport for Mautic 7 (v1.1.0) with a Mautic-4-style retry state machine, middleware-owned temporary retries, recovery, backoff, keepalive, updated docs, and a full PHPUnit + concurrent race-condition test suite (CI on PHP 8.2 / 8.5 / 8.6).

Root cause fixed

Commit d057a771 introduced listMessageIdsGenerator() which yielded basenames only. Downstream code treated those values as full paths → messages were not created/read under the correct var/queue/… location.

This PR replaces that design with an ID-centric claim path: ready IDs are listed, full paths are always built via helpers, and claim is a single atomic rename().

State machine

Stage Filename Actor
First enqueue {id}.message send()
Claim {id}.processing tryClaim()
Temporary fail rename → {id}.message.retry.N Middleware + requeueWithRetry() → Worker ack()
Claim after retry {id}.processing.retry.N tryClaim()
Final fail file deleted Worker reject() → failed transport
Stuck recovery .processing* → ready (same attempt) recoverStuckProcessingFiles()

Guarantees:

  • A message is never in both the active queue and the failed queue at the same time
  • Temporary failures do not create a second copy (same file is renamed back to ready)
  • Concurrent workers cannot double-claim the same message (atomic rename(); loser gets null)

Main changes

New

  • Messenger/Stamp/AttemptStamp.php
  • Messenger/Middleware/FilesystemRetryMiddleware.php
  • PHPUnit suite (tests/Unit/…, tests/Concurrency/…)
  • GitHub Actions workflow (.github/workflows/tests.yml)
  • phpunit.xml.dist, tests/bootstrap.php, standalone CoreParametersHelper stub for CI

Updated

  • Messenger/Transport/FileSystemTransport.php — hardened rewrite
  • Messenger/Transport/FileSystemTransportFactory.php — cleaned (no unused PathsHelper)
  • Config/services.php — registers middleware + transport factory
  • Config/config.php — version 1.1.0 + retry/recovery parameters
  • composer.json — require-dev + autoload-dev
  • README.md — state machine, settings tables, testing, architecture

New config parameters

'filesystem_queue_max_attempts'        => 5,
'filesystem_queue_recovery_timeout'    => 300,  // seconds
'filesystem_queue_recovery_interval'   => 30,
'filesystem_queue_recovery_stuck'      => true,
'filesystem_queue_retry_backoff_base'  => 30,   // exponential seconds
'filesystem_queue_send_max_retries'    => 15,

…chine

- Fix path/iterator breakage from d057a77 (listMessageIdsGenerator + LimitIterator)
- Introduce AttemptStamp and FilesystemRetryMiddleware for temporary retries
- Atomic claim via rename; .message / .message.retry.N / .processing / .processing.retry.N
- Backoff, recovery, keepalive, no dual-queue leakage
- Updated factory, services.php, config parameters
- Version bump to 1.1.0
- PHPUnit tests for FileSystemTransport, FilesystemRetryMiddleware, AttemptStamp, Factory
- GitHub Actions matrix: PHP 8.2 / 8.5 / 8.6
- phpunit.xml.dist + tests/bootstrap.php
- composer.json require-dev + autoload-dev
- Add require-dev (phpunit, symfony components)
- Add autoload-dev for tests
- Provide minimal CoreParametersHelper stub so tests run without full Mautic core-lib
Anonymous classes must implement MiddlewareInterface for Symfony 7 StackMiddleware.
- Concurrent claim: N workers, M messages → each message claimed exactly once
- Concurrent send: unique IDs, no lost/dupe files
- Concurrent requeue + claim: no double-processing after retry
- Workers spawned as real PHP CLI processes sharing one queue dir
@wieslawgolec
wieslawgolec merged commit 0d9e070 into mautic7.x Oct 5, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant