Skip to content

feat: self-provisioning Gmail connector (metadata-only, two-stage sync→build) - #2

Open
maxdolphin wants to merge 16 commits into
base/pre-gmail-connectorfrom
feat/gmail-connector
Open

feat: self-provisioning Gmail connector (metadata-only, two-stage sync→build)#2
maxdolphin wants to merge 16 commits into
base/pre-gmail-connectorfrom
feat/gmail-connector

Conversation

@maxdolphin

@maxdolphin maxdolphin commented Jul 6, 2026

Copy link
Copy Markdown
Owner

Summary

A self-provisioning Gmail connector that lets a Google Workspace admin connect their org once and have OASIS build a weighted communication-flow network from message metadata, precompute it, and open it in the existing analysis view — no manual CSV export.

Base note: this PR is based on base/pre-gmail-connector (the pre-connector state of feat/detailed-ecosystemic-report) rather than main, because the connector builds on the M2 self-service ingestion primitive (src/network_ingestion.py) which is not yet on main. This keeps the diff connector-only for review.

Architecture — two-stage (approach B)

  • Stage 1 — GmailConnector.sync() (src/connectors/gmail_connector.py): admin OAuth via domain-wide delegation, least-privilege gmail.metadata scope, pulls From/To/Cc/timestamp/thread/size into a persisted, idempotent gmail_interactions SQLite table. Metadata-only — never subject or body.
  • Stage 2 — build_flow_matrix() (src/connectors/gmail_weighting.py): a pure, clock-free transform — hybrid recency-decay × sustained-engagement weighting, external-address filtering, individual or department granularity — feeding the existing build_flow_matrix_from_edges → provision_network → get_full_profile path.
  • UI (app.py): 🔌 Connect Gmail mode (connect → sync → build → analyze).
  • Retires the legacy GoogleWorkspaceConnector.get_flow_data stub (it silently returned a zero matrix → now refuses loudly and points to the new connector).

Privacy & scientific integrity

  • Metadata-only enforced in depth: gmail.metadata scope + format='metadata' + a schema with no subject/body columns (guarded by a test).
  • Non-negative flows guaranteed: β/half-life validated (ValueError on β<0, half_life<=0); no core Ulanowicz formula is touched. β and half-life are documented calibration parameters for network construction, not metric formulas.
  • Idempotent inserts (INSERT OR IGNORE on a message-edge UNIQUE index) so re-syncing overlapping windows can't double-count flows.
  • Clock purity: reusable modules never read the wall clock; now_utc is always passed in.

Testing

  • ~22 new connector tests (TDD): store round-trip + metadata-only schema; weighting decay/sustained/granularity/external-filter/window/guards; connector auth/sync/idempotency/address-parsing.
  • Full tests/ suite: 321 passed, zero regressions.
  • UI smoke-tested live (privacy notice, no-credentials warning, empty-window guard).

Notes / follow-ups

  • A real Gmail pull needs a service account with domain-wide delegation configured in .streamlit/secrets.toml; all logic is testable without it via injected fake clients.
  • Slack / Microsoft 365 are deliberately out of scope — the BaseConnector / ConnectorFactory seams are reserved for them.
  • Rate-limit backoff for very large domains is a documented follow-up in the real-API adapter.

🤖 Generated with Claude Code

maxdolphin and others added 16 commits July 6, 2026 18:11
@maxdolphin @claude
docs(spec): Gmail connector design (approach B, two-stage sync/build)
7b05076 
Self-provisioning Gmail network source: admin OAuth (domain-wide
delegation, metadata-only gmail.metadata scope), org-wide pull into a
persisted gmail_interactions table (Stage 1), and a pure re-runnable
weighting/build step (Stage 2) feeding the existing
build_flow_matrix_from_edges -> provision_network -> get_full_profile
path. Hybrid recency-decay x sustained-engagement weighting, both
individual and department granularity, configurable window + half-life.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@maxdolphin @claude
docs(spec): tighten Gmail connector spec after review
32fe61f 
- §5.1 node resolution: individual=email, department=orgUnitPath leaf;
  filter external addresses to known org users (drop + report count)
- BaseConnector conformance: explicit sync()/build() + get_flow_data wrapper
- sync_run_id / now_utc minted in UI layer, never in reusable modules
- add external-filtering test to the plan surface

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@maxdolphin @claude
docs(plan): Gmail connector implementation plan (7 tasks, TDD)
65f404b 
Bite-sized TDD tasks for the two-stage Gmail connector: package skeleton,
metadata-only SQLite store, pure hybrid weighting (Stage 2), GmailConnector
auth+sync (Stage 1), legacy-stub retirement, Connect Gmail UI, full-suite
regression. Each task ships tests + a commit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@maxdolphin @claude
feat(connectors): package skeleton + Google client deps
a3f5bb3 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@maxdolphin @claude
feat(connectors): gmail_interactions store (metadata-only DAO)
2ff0f35 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@maxdolphin @claude
fix(connectors): lazy package exports (PEP 562) so submodules import …
e13f851 
…standalone

Eager imports in __init__ coupled all three submodules — importing gmail_store
triggered imports of not-yet-created siblings, forcing throwaway stubs and making
intermediate commits non-standalone. Resolve exports lazily via __getattr__ so
each module imports independently and every commit stays green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@maxdolphin @claude
refactor(connectors): address store review — drop dead _COLUMNS, expl…
b3d326c 
…icit conn close, more tests

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@maxdolphin @claude
feat(connectors): pure hybrid decay x sustained weighting (Stage 2)
983c2cf 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@maxdolphin @claude
refactor(connectors): validate weighting inputs; narrow import except…
9ebc3f3 
…; doc fixes

Guard half_life<=0 / beta<0 / window<0 (negative beta would inject negative
flow weights, invalid in the Ulanowicz model). ImportError-only fallback.
Correct 'ISO weeks' -> 7-day epoch windows in docstring.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@maxdolphin @claude
feat(connectors): GmailConnector auth + metadata sync (Stage 1)
bef4451 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@maxdolphin @claude
fix(connectors): idempotent inserts + atomic auth + robust address pa…
757dbe8 
…rsing

Store-level INSERT OR IGNORE on message-edge identity so re-syncing overlapping
windows no longer double-counts flows (would inflate Ulanowicz weights). Atomic
client assignment in authenticate. RFC 2822 address parsing via email.utils.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@maxdolphin @claude
refactor(connectors): retire GoogleWorkspace stub, point to GmailConn…
89eebd2 
…ector

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@maxdolphin @claude
feat(app): Connect Gmail data-source mode (sync + build + analyze)
812d6c5 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@maxdolphin @claude
fix(app): guard st.secrets access in Connect Gmail (no secrets.toml c…
4d7b5d8 
…rash)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@maxdolphin @claude
docs(spec): mark Gmail connector implemented
bca221a 
Full tests/ suite green: 321 passed (incl. ~22 new connector tests).
Pre-existing validation/test_florida_bay.py collection error (relative-path
script assuming CWD=validation/) is unrelated and out of scope.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@maxdolphin @claude
fix(connectors): metadata-scope-safe SENT listing, per-user sync isol…
84a01b7 
…ation, UI guard

- gmail.metadata scope forbids the q search param (403); list SENT + filter
  internalDate client-side, preserving metadata-only.
- per-user try/except so one bad mailbox doesn't abort the whole sync.
- guard build_flow_matrix's NetworkIngestionError in the UI (empty/external-only
  window) with a friendly warning instead of a traceback.
- replace deprecated datetime.utcnow() with datetime.now(timezone.utc).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@maxdolphin