Backup/restore corrupts non-UTF8 RFC822 content (binary attachments lost)

[FernandoCelmer]

Description

The sync pipeline decodes each fetched RFC822 message into a Python str using UTF-8 with errors="replace", then stores it as the file column of RawSerializer. Restore re-encodes the same string with raw.file.encode("utf-8"). Any byte that is not valid UTF-8 — common in base64-encoded binary attachments, latin-1 bodies, or 8-bit MIME parts — is replaced by U+FFFD on the way in and cannot be reconstructed on the way out. The library advertises itself as a backup/restore tool, so this is a data-loss bug, not a cosmetic one.

Location

• email_profile/clients/imap/sync.py line ~195 (file=entry.text())
• email_profile/clients/imap/parser.py line ~74 (FetchParser.text() calls _decode(self._body))
• email_profile/clients/imap/mailbox.py line ~122 (raw = message.file.encode("utf-8") on APPEND)
• email_profile/serializers/raw.py line ~15 (file: str)

Current Behavior

# sync.py
raw = RawSerializer(
    message_id=entry.message_id,
    uid=entry.uid,
    mailbox=mailbox.name,
    flags=entry.flags,
    file=entry.text(),   # bytes -> str with errors="replace"
)

# parser.py
@staticmethod
def _decode(data: bytes, errors: str = "replace") -> str:
    if isinstance(data, bytes):
        return data.decode("utf-8", errors=errors)
    return str(data)

# mailbox.py
elif isinstance(message, str):
    raw = message.encode("utf-8")

Any non-UTF8 byte sequence in the original message is lossily replaced before storage. The original bytes are unrecoverable.

Expected Behavior

Store the original RFC822 bytes verbatim. Restore should upload the same bytes that were fetched, byte-for-byte.

Suggested Fix

Change RawSerializer.file to bytes, propagate bytes through Fetch / Sync / Restore, and use a LargeBinary (BLOB) column in RawModel. Example:

# raw.py
class RawSerializer(BaseModel):
    message_id: str
    uid: str
    mailbox: str
    flags: str = ""
    file: bytes

# sync.py
raw = RawSerializer(..., file=entry.raw())   # bytes, not text()

# mailbox.py append()
elif isinstance(message, (bytes, bytearray)):
    raw = bytes(message)

# models/raw.py
file: Mapped[bytes] = mapped_column(LargeBinary)

Migration: existing rows can be re-encoded as latin-1 (round-trips any byte sequence) before column type change.

Impact

• Binary attachments (PDFs, images, archives) re-uploaded after restore are corrupted.
• Messages with latin-1, windows-1252, or other non-UTF8 bodies lose characters.
• Hash-based deduplication of restored messages differs from server-side hashes.

Since the library is marketed as the simplest way to back up and restore mailboxes, this defeats the primary use case for any non-trivial inbox.

Priority

Critical

[FernandoCelmer]

Fixed in #80 (merged to develop).