Decode RFC 2047 display names at the raw .eml boundary and pin the resynced SwiftMail fork

Companion/Memory/Current/126-swiftmail-upstream-sync-and-oversized-train.md

@@ -0,0 +1,73 @@
+# SwiftMail upstream synchronization and the parked oversized-metadata train
+
+## Status checked 2026-09-09
+
+SwiftMail upstream main is `64a9a861de7558733cae4e51be6fedcdb128dc4f`. Synchronization to this candidate is BLOCKED by reproduced receive-path compatibility regressions; neither fork main nor the app pin was changed.
+
+The three contributions on the old combined fork pin `3a904d8a5257162cc3935ab0090bc94333dc8022` were incorporated upstream on September 7:
+
+- Header/address encoding: #215 → #226.
+- MIME parameter encoding/parsing: #216 → #230.
+- Validated partial IMAP fetching: #219 → #228.
+
+The replacement branches retained our original commits and added maintainer fixes. SwiftMail's latest published release is 1.11.0, which predates those merges. The app and notification extension share the exact `packages.SwiftMail.revision` in `project.yml`; changing the fork alone does not change the app dependency.
+
+## Separate upstream release dependency
+
+Draft iOS #103 and issue #74 remain parked. Apple swift-nio-imap #849 merged on September 9 as `26a69caf8487bc91da4fd10aaa119776b526e18c`, exposing `IMAPClientHandler.maximumBufferSize`. The latest tag, 0.4.0, does not contain this commit. SwiftMail at the candidate above still calls `IMAPClientHandler(parserOptions:)` without the new argument and retains its existing literal-limit restriction.
+
+Wait for a containing NIO IMAP release, then SwiftMail adoption and effective limit wiring with fragmented-response tests. Do not file a duplicate implementation of #849 or treat partial-body fetching as a solution to the separate non-streaming metadata limit. The #849 description mentions a 1 MiB default, but the inspected source still defaults through `IMAPDefaults.lineLengthLimit`, defined as 8,192; use source and executable evidence when integrating.
+
+Issue #10's header/MIME acceptance criterion is satisfied; its app synchronization and verification criteria remain separate. Issue #9 concerns pipelined-FETCH promise cleanup, not this NIO IMAP release dependency.
+
+## Migration and resume boundary
+
+Do not reserve a migration number or edit migration code while parked. When #103 resumes, choose the next free number from then-current main, update migration tests and the foreign-key-mode census, and account for development databases that ran the old draft before launching it. Earlier instructions prescribing a specific replacement number are superseded. No migration execution, reset, rebase, or release is part of the housekeeping scope.
+
+Reconcile the shipped #105 `bodyMetadataOversized` admission state with the bounded-fetch recovery path. Reassess both metadata-overflow catches and the part-fetch classifier: fragmentation-dependent metadata overflow is not a deterministic unsupported-part response. Fresh integration validation is required before #103 becomes ready.
+
+## Sources
+
+- https://github.com/Cocoanetics/SwiftMail/pull/226
+- https://github.com/Cocoanetics/SwiftMail/pull/230
+- https://github.com/Cocoanetics/SwiftMail/pull/228
+- https://github.com/apple/swift-nio-imap/pull/849
+- https://github.com/TabMail/tabmail-ios/pull/103
+- https://github.com/TabMail/tabmail-ios/issues/74
+- https://github.com/TabMail/tabmail-ios/issues/10
+
+## Candidate validation and hold
+
+The independent dependency review covered all twelve incoming commits. Architecture was clean; correctness, robustness/security, and test coverage were unclean. No clean gate is claimed. SwiftMail build and strict lint passed (zero lint violations); 619 tests in 77 suites passed. Four CLI logging deprecation diagnostics also occur on the exact baseline. No iOS app/NSE build or device test was run because the collision check blocked adoption.
+
+Reproduced against the exact old app pin and candidate, using unchanged app parsing/rendering helpers:
+
+- Parsed raw `.eml` display names now remain RFC 2047 encoded in app output. Message addresses and body remain intact, so this is a display/index-text representation collision, not a wrong-recipient claim.
+- Extended filenames with omitted charset or ASCII values labeled Windows-1252/ISO-8859-15 are discarded and become generated names; attachment bytes remain intact.
+- MIME continuation parsing repeatedly scans the full header for each numbered segment. An optimized synthetic 72,635-byte header took 8.08 seconds versus 0.031 seconds on the baseline; a 146,363-byte case exceeded 25 seconds. Full MIME parsing was separately shown to reach the expensive path. Device timing was not measured.
+
+The coordinator independently reran the name/filename assertions (baseline passes, candidate fails; payload/address/subject controls pass) and the 1,024-segment optimized workload (candidate approximately 0.48 seconds versus baseline 0.007 seconds). Retrying successful parsing does not repair either metadata loss or excessive work. Holding the dependency advance preserves the current consumer behavior and the draft's existing partial-fetch API without introducing a fork deviation.
+
+Further acceptance-test gaps were reported for the actual installed reconnect callback and SMTP refusal-reason collection. These are coverage gaps, not reproduced runtime failures. Keep current pins pending a decision on focused upstream fixes and any required app display adaptation; do not advance the fork based solely on the passing library suite.
+
+## Fixes prepared 2026-09-09 (owner decision: upstream PR for the library bugs, app-side decode for the display-name change)
+
+Root causes, from the B→C source diff rather than the report's symptoms:
+
+- **R1 (quadratic continuation parsing) and C2 (lost filenames) are both regressions introduced by upstream #230**, the maintainer's parameter-boundary hardening. Neither exists at the pinned baseline `a2d4a94f`, which had no RFC 2231 continuation support at all and decoded `filename*` with no charset check. #230's new `extendedContinuation` looked each numbered section up with `extractHeaderParam`, which re-tokenizes the whole header per call (N sections → ~2N full scans). Its new `decodeExtendedBytes` whitelisted `utf-8`/`us-ascii`/`iso-8859-1` and returned nil for every other label including the blank one that RFC 2231 §4 explicitly permits. The tests added with #230 cover only UTF-8-labelled happy paths and the injection cases, so the maintainer's suite stays green.
+- **C1 (encoded display names) is a deliberate contract change in upstream #226**, not a bug: parsed addresses stay in header wire form. The app must decode at its own boundary.
+
+What was done:
+
+- **SwiftMail fix**, branch `fix/rfc2231-extended-parameters` on the fork, based on upstream `64a9a86`, one commit: the header is tokenized once into `EMLParser.parameters(of:)` and both the lookup and the continuation collector read from that list; RFC 2231 §3 section rules (decimal from 0, no leading zeroes, no gaps, first duplicate wins) are applied; a blank charset decodes as UTF-8 and any other label resolves through Foundation's `String.Encoding(ianaCharsetName:)`, the lookup the encoded-word decoder already uses; unknown labels still yield nil and fall through to the literal spelling. The RFC 2231 helpers moved to `EMLParser+RFC2231.swift` for the lint file-length limit. 624 tests pass; strict lint clean; the blank-charset, platform-charset and 4,096-section linear-work tests are red on unmodified upstream (the last after 34 s in a debug build). Upstream PR target: `Cocoanetics/SwiftMail`, head `TabMail:fix/rfc2231-extended-parameters`.
+- **App-side C1 fix** in `EmlParsing.parse`: `RFC5322Parse.decodeRFC2047` is applied to subject, from, to and cc before they reach `EmlMarker`. The decoder is a no-op on text without an encoded-word, so it is correct against both the current pin (which decodes in the library) and the candidate (which does not). Consumer test `parseDecodesEncodedDisplayNames` in `EmlParsingTests`.
+- **Red/green for C1 against the fixed candidate** was reproduced with the housekeeping consumer-probe harness (`LOCAL_SWIFTMAIL_HOUSEKEEPING_20260909/compile-probe.py`, compiled against the fixed SwiftMail build): with the pre-fix `EmlParsing`, `htmlContainsDecodedNames` is false and From/To/Cc carry `=?UTF-8?B?…?=`; with the fixed `EmlParsing`, all names decode and addresses are retained. In the same run every `filename*` charset variant (`UTF-8`, `us-ascii`, `iso-8859-1`, `windows-1252`, `iso-8859-15`, blank) projects `invoice.pdf`, closing C2. Evidence: `LOCAL_SWIFTMAIL_HOUSEKEEPING_20260909/fix-evidence/` (local, excluded from git).
+
+Still required before the pin advances: upstream merge of the fix (or a fork-local deviation carrying it, which the sync skill then replays on every resync), then the fork-sync skill's reset/re-pin, then the iOS validations listed in the housekeeping report (app + NSE build via `Scripts/xcodegen.sh`, the consumer suites, real-provider EHLO acceptance for #229). `IOS-IMAP-016`'s "when revisited" items 1–3 appear to be delivered by #226/#230; verify against the final pinned source before closing that record.
+
+## Sync executed 2026-09-09 (owner: carry the fix as a fork deviation, pin, merge)
+
+- Fork `main` was reset to upstream `64a9a86` plus the fix commit and force-pushed: **fork `main` = `d87295e3e48b014ec7c225a68012186fddaff15a`**, identical to branch `fix/rfc2231-extended-parameters`. `git log upstream/main..main` is exactly ONE commit — the fork now carries a deviation again. Identify it by its subject ("Read RFC 2231 continuations in one pass and honor every charset the platform names"), not its hash; the next resync must replay it until upstream merges it, then drop it.
+- Upstream PR: https://github.com/Cocoanetics/SwiftMail/pull/233 (head `TabMail:fix/rfc2231-extended-parameters`). When it merges, upstream's version of the change supersedes the deviation.
+- `project.yml` `packages.SwiftMail.revision` → `d87295e3e48b014ec7c225a68012186fddaff15a` (app and NSE share it). Resolved checkout proven by `git -C <spm>/checkouts/SwiftMail rev-parse HEAD` before and after the build. App, NSE and TabMailTests built with only the pre-existing `AutoSizingHTMLViewRevealSeedingTests` deprecation warning.
+- The old fork main `3a904d8` (our #215/#216/#219 originals) is gone from the fork; those changes live upstream as #226/#230/#228.

Companion/Process/Current/KnownIssues/Amendments/ios-imap-016.md

@@ -1,5 +1,7 @@
 # IOS-IMAP-016 — SwiftMail's header encoder has the same inverted guard, and its part-header builder interpolates filenames raw
 
+> **2026-09-09 status amendment:** upstream acceptance is complete through SwiftMail #226 and #230 (superseding #215 and #216). App/fork synchronization and regression verification remain pending; this record stays open. The historical defect and pin descriptions below are retained. See [current sync and release-train status](../../../../Memory/Current/126-swiftmail-upstream-sync-and-oversized-train.md).
+
 **Class:** `open` · **Opened:** 2026-08-12 · **Remedy is UPSTREAM** (Cocoanetics/SwiftMail, not the
 TabMail fork — per the repo convention that SwiftMail PRs go upstream).
 **Fully mitigated at TabMail's app boundary by PR #37**, so these are no longer live Subject defects

PROJECT_MEMORY.md

@@ -169,3 +169,4 @@ Authored after `v1.6.38`, so deliberately **not** rows in [`manifest.tsv`](Compa
 | Current | 🚨 **A DURABLE WRITE TO A MIRRORED IDENTITY COLUMN MUST REFRESH THE NSE MIRRORS** — `nse.accountMap`/`nse.imapAccounts` are the extension's ONLY resolver; `addIMAPAccount`/`addICloudAccount` refreshed neither, so `handleIMAPReconnect` early-returned; `mirrorAccountIdentity()`; `IOS-NSE-008` | [read in full](Companion/Memory/Current/123-a-durable-write-to-a-mirrored-identity-column-must-refresh-the-nse-mirrors.md) |
 | Current | 🚨 **ORDINARY SIGN-OUT RELEASES THE DEVICE PUSH REGISTRATION AND ENDS THE AUTH SESSION** — `TabMailAuthService.signOut()`: `unregisterDeviceForSignOut()` FIRST, then GoTrue `logout?scope=local`, **BOTH legs always**; `guard getSession()?.userId == subject`; push#42; `SignOutHandshakeTests` | [read in full](Companion/Memory/Current/124-ordinary-sign-out-releases-the-device-push-registration-and-ends-the-auth-session.md) |
 | Current | Background credential refresh shares grant ownership — `ProviderCredentialStore`, `CredentialRefreshGate`, `TabMailSessionRefresh`; NSE cancellation, rotation, and foreground repair (#129) | [read in full](Companion/Memory/Current/125-background-refresh-shares-grant-ownership.md) |
+| Current | SwiftMail upstream sync, exact app/NSE revision, #215/#216/#219 replacements, NIO IMAP #849 release dependency, parked #103/#74 and migration numbering on resume; 2026-09-09 fixes — R1 quadratic RFC 2231 continuations + C2 blank/windows-1252 charset filename loss are upstream #230 regressions fixed on fork branch `fix/rfc2231-extended-parameters` (upstream PR), C1 encoded display names (#226 contract) decoded app-side in `EmlParsing.parse` via `RFC5322Parse.decodeRFC2047` | [read in full](Companion/Memory/Current/126-swiftmail-upstream-sync-and-oversized-train.md) |

Shared/Parse/EmlParsing.swift

@@ -51,12 +51,22 @@ enum EmlParsing {
     static func parse(rawBytes: Data) -> Parsed? {
         guard let message = try? EMLParser.parse(rawBytes) else { return nil }
 
+        // SwiftMail decodes the Subject itself but hands address strings back
+        // in header wire form: a display name that could not be written
+        // literally stays an RFC 2047 encoded-word (`=?UTF-8?B?...?=`), so the
+        // addr-spec is identified before any decoding can turn name text into
+        // address syntax. `EmlMarker` escapes HTML but does not decode MIME, so
+        // the names are decoded here with SwiftMail's own `decodeMIMEHeader`,
+        // the decoder that produced the Subject: it carries the full charset
+        // table and does bounded work on malformed input. The Subject is passed
+        // through untouched; decoding it a second time would turn a literal
+        // encoded-word-shaped subject into its decoded form.
         let envelope = EmlMarker.Envelope(
             subject: message.subject,
-            from: message.from,
+            from: message.from?.decodeMIMEHeader(),
             date: message.date,
-            to: message.to,
-            cc: message.cc
+            to: message.to.map { $0.decodeMIMEHeader() },
+            cc: message.cc.map { $0.decodeMIMEHeader() }
         )
 
         let bodyHtml: String

TabMailTests/Shared/EmlParsingTests.swift

@@ -47,6 +47,96 @@ struct EmlParsingTests {
         #expect(parsed.bodyHtml.contains("HELLO FROM HTML BODY"))
     }
 
+    @Test("parse decodes RFC 2047 display names in From/To/Cc for rendering")
+    func parseDecodesEncodedDisplayNames() throws {
+        // `=?UTF-8?B?Sm9zw6k=?=` is "José". SwiftMail may hand the name back
+        // in wire form; TabMail's envelope must carry the decoded text, since
+        // EmlMarker renders and indexes it without any further MIME decoding.
+        let rfc822 = """
+        From: =?UTF-8?B?Sm9zw6k=?= <sender@example.com>\r
+        To: =?UTF-8?Q?Ren=C3=A9e?= <to@example.com>, Plain <plain@example.com>\r
+        Cc: =?UTF-8?B?7ZWc7IaU?= <cc@example.com>\r
+        Subject: =?UTF-8?B?7ZWc7IaU?= subject\r
+        Date: Wed, 2 Oct 2025 01:50:00 +0000\r
+        Content-Type: text/plain; charset=utf-8\r
+        \r
+        body
+        """
+        let parsed = try #require(EmlParsing.parse(rawBytes: Data(rfc822.utf8)))
+
+        let from = try #require(parsed.envelope.from)
+        #expect(from.contains("José"))
+        #expect(from.contains("sender@example.com"))
+        #expect(!from.contains("=?"))
+        #expect(parsed.envelope.to.contains { $0.contains("Renée") && $0.contains("to@example.com") })
+        #expect(parsed.envelope.to.contains { $0.contains("Plain") && $0.contains("plain@example.com") })
+        #expect(parsed.envelope.cc.contains { $0.contains("한솔") && $0.contains("cc@example.com") })
+        #expect(parsed.envelope.subject == "한솔 subject")
+        #expect(!parsed.envelope.to.contains { $0.contains("=?") })
+        #expect(!parsed.envelope.cc.contains { $0.contains("=?") })
+
+        // The decoded name reaches the rendered marker text, not only the envelope.
+        let html = EmlMarker.build(filename: "n.eml", partSection: "1", envelope: parsed.envelope, bodyHtml: parsed.bodyHtml)
+        #expect(html.contains("José"))
+        #expect(!html.contains("=?UTF-8?"))
+    }
+
+    @Test("parse keeps a literal encoded-word-shaped subject and decodes legacy-charset names")
+    func parseDecodesLegacyCharsetNamesAndKeepsLiteralSubject() throws {
+        // The Subject below is the OUTER encoding of the literal text
+        // `=?UTF-8?B?Sm9l?=` (what `RFC2047.encodeHeaderValue` emits for a
+        // subject that merely looks like an encoded-word). SwiftMail decodes
+        // that outer layer once; the app must not decode the literal again.
+        // The names use charsets outside the minimal app decoder's table.
+        let rfc822 = """
+        From: =?windows-1252?B?Sm9z6Q==?= <sender@example.com>\r
+        To: =?windows-1252?Q?Jos=E9?= <to@example.com>\r
+        Cc: =?ISO-8859-2?Q?=A3ukasz?= <cc@example.com>\r
+        Subject: =?UTF-8?B?PT9VVEYtOD9CP1NtOWw/PQ==?=\r
+        Date: Wed, 2 Oct 2025 01:50:00 +0000\r
+        Content-Type: text/plain; charset=utf-8\r
+        \r
+        body
+        """
+        let parsed = try #require(EmlParsing.parse(rawBytes: Data(rfc822.utf8)))
+
+        #expect(parsed.envelope.subject == "=?UTF-8?B?Sm9l?=")
+        #expect(parsed.envelope.from == "José <sender@example.com>")
+        #expect(parsed.envelope.to == ["José <to@example.com>"])
+        #expect(parsed.envelope.cc == ["Łukasz <cc@example.com>"])
+        #expect(parsed.bodyHtml.contains("body"))
+
+        let metadata = EmailFilter.parseEmlSectionMetadata(
+            html: EmlMarker.build(filename: "l.eml", partSection: "1", envelope: parsed.envelope, bodyHtml: parsed.bodyHtml),
+            filename: "l.eml"
+        )
+        #expect(metadata?.subject == "=?UTF-8?B?Sm9l?=")
+        #expect(metadata?.from == "José <sender@example.com>")
+    }
+
+    @Test("parse does bounded work on an oversized malformed Q-encoded name")
+    func parseBoundedWorkOnOversizedMalformedName() throws {
+        // A sender controls the header bytes of an attached .eml. One huge
+        // Q-encoded name (far past RFC 2047's 75-octet word limit) must not
+        // turn envelope decoding into quadratic work; the parse must finish
+        // quickly and still yield the decoded name and the body.
+        let escapes = 16_384
+        let name = "=?UTF-8?Q?" + String(repeating: "=41", count: escapes) + "?="
+        let rfc822 = "From: \(name) <sender@example.com>\r\nTo: Plain <to@example.com>\r\nSubject: S\r\nDate: Wed, 2 Oct 2025 01:50:00 +0000\r\nContent-Type: text/plain; charset=utf-8\r\n\r\nbody"
+
+        let clock = ContinuousClock()
+        var parsed: EmlParsing.Parsed?
+        let elapsed = clock.measure {
+            parsed = EmlParsing.parse(rawBytes: Data(rfc822.utf8))
+        }
+        let result = try #require(parsed)
+        let from = try #require(result.envelope.from)
+        #expect(from.hasSuffix(" <sender@example.com>"))
+        #expect(from.filter { $0 == "A" }.count == escapes)
+        #expect(result.bodyHtml.contains("body"))
+        #expect(elapsed < .seconds(1))
+    }
+
     @Test("parse text/plain message returns plainTextToHTML-converted body")
     func parsePlainTextBody() throws {
         let rfc822 = """

project.yml

@@ -35,7 +35,7 @@ settings:
 packages:
   SwiftMail:
     url: https://github.com/TabMail/SwiftMail.git
-    revision: a2d4a94f844db62843ef6aec16f3ed9462152acc
+    revision: d87295e3e48b014ec7c225a68012186fddaff15a
   SwiftSoup:
     url: https://github.com/scinfu/SwiftSoup.git
     from: "2.6.0"