feat(warehouse): add Shipmail source

[jcoulaud]

Problem

Shipmail users cannot sync email operations data into PostHog's Data warehouse. Shipmail now exposes an analytics-safe API for message activity, but PostHog has no source for it.

Changes

• Add an alpha Shipmail warehouse source for message analytics, mailboxes, domains, and suppressions.
• Sync messages incrementally in stable updated_at order and resume every table from Shipmail's opaque pagination cursor.
• Validate Bearer credentials and per-table scopes through one /capabilities request.
• Add canonical table descriptions, the official Shipmail mark, generated source configuration and schema entries, and a choices-only Django migration.

Before:

flowchart LR
    A[Shipmail API] --> B[No PostHog source]
    classDef phRed fill:#f54e00,stroke:#f54e00,color:#fff;
    classDef phGray fill:#e5e7eb,stroke:#c7ccd1,color:#000;
    class A phRed;
    class B phGray;

Loading

After:

flowchart LR
    A[Shipmail API] --> B[Tracked REST source] --> C[Resumable import pipeline] --> D[Data warehouse]
    classDef phBlue fill:#1d4aff,stroke:#1d4aff,color:#fff;
    classDef phRed fill:#f54e00,stroke:#f54e00,color:#fff;
    classDef phYellow fill:#f9bd2b,stroke:#f9bd2b,color:#000;
    class A phRed;
    class B,C phBlue;
    class D phYellow;

Loading

How did you test this code?

• hogli test products/warehouse_sources/backend/temporal/data_imports/sources/shipmail – 18 tests cover pagination and resume checkpoints, incremental watermarks, response-shape failures, table metadata, credential validation, and per-table scope reporting.
• hogli test products/warehouse_sources/backend/temporal/data_imports/sources/tests – 3,851 source-registry and generated-config tests pass.
• ruff check, ruff format --check, ty check, and focused mypy --follow-imports=skip pass for the new package.
• pnpm run schema:build, pnpm run generate:source-configs, and hogli ci:preflight --strict pass.
• python manage.py makemigrations warehouse_sources --check --dry-run reports no model drift. I did not apply the choices-only migration locally.

A full focused mypy graph also reached an existing error in posthog/settings/managed_migrations.py:30; it reported no Shipmail error before stopping. The focused check with imported modules skipped passes.

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Automatic notifications

• Publish to changelog?
• Alert Sales and Marketing teams?

Docs update

PostHog/posthog.com#19086 adds setup, scope, sync-mode, privacy, and troubleshooting guidance.

🤖 Agent context

Autonomy: Human-driven (agent-assisted)

Codex implemented the source in a user-directed desktop task; no public session link is available. Fable reviewed the prerequisite Shipmail API contract before implementation.

Repo skills used: implementing-warehouse-sources, documenting-warehouse-sources, django-migrations, writing-tests, writing-user-facing-copy, and writing-code-comments. The source keeps PostHog's unversioned API sentinel even though Shipmail's stable route prefix is /api/v1, uses the shared tracked REST transport, and checks all table scopes from one capabilities response.

GitHub does not allow this external contributor account to self-assign the upstream PR, so it remains unassigned for a PostHog maintainer to route.

[trunk-io]

Merging to master in this repository is managed by Trunk.

• To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

[jcoulaud]

Maintainer handoff:

• Rebased onto current master at 066e4e2fc1; explicit base comparison reports 0 commits behind after the push.
• Standardized the product name and enum value to Shipmail.
• Regenerated the choices migration as 0116_shipmail_source after upstream 0115_scaffold_four_requested_sources; the migration was not applied.
• Removed the two empty __init__.py files.
• Re-ran 18 focused source tests, 3,862 registry/config/category/version tests, Ruff, formatting, migration drift detection, and strict preflight. All relevant checks pass locally. The new isolated source-loader behavior and shared pipeline changes ran 46 additional tests successfully; 11 database-backed tests could not start because the local PostgreSQL service is unavailable.
• There are no unresolved review threads. Security checks are running again on the rebased head.

GitHub requires maintainer approval for the fork-triggered workflows, and this external contributor account cannot approve them or request reviewers. @PostHog/team-warehouse-sources, could someone approve the workflows and review this PR? The companion docs PR is PostHog/posthog.com#19086.