Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 19 additions & 13 deletions SESSION_HANDOFF.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,16 @@
# Session Handoff

## 2026-07-24 — Exact body completion percentages

- Branch `fedster99/fix-body-progress-rounding` makes 100% an exact completion
claim for live and priority account progress and per-folder current body
progress. Incomplete ratios are floored, so 199 fetched bodies out of 200
reports 99%, not 100%.
- Public migration `0023_exact_body_completion_percentages` replaces
`imap_account_progress`; the repository applies the same rule to folder rows.
A real-Postgres Scenario R regression covers priority, live, and folder
progress at the rounding boundary.

## 2026-07-23 — Content extract and body-store seam

- Branch `fedster99/content-body-store-seam` adds public migration
Expand Down Expand Up @@ -165,20 +176,16 @@
Commit/merge, immutable-image publication, and downstream production re-pin
are the next actions.

Last updated: 2026-07-23
Last updated: 2026-07-24

This is the tracked restart point for future agents. Keep it concise, factual, and safe to publish. Put private local notes, credentials, customer/provider probes, and one-off scratch work in `.context/` instead.

## Current Branch

- Active branch `fedster99/content-body-store-seam` implements ADR 0028's
evidence-first content seam. The cloud image re-pin is deliberately deferred
to a separate private-repo task.
- Active branch `fedster99/live-body-coverage-policy` makes
`body_fetch_policy` mutable for existing accounts and replaces counter-derived
live/priority body coverage with current-row evidence. ADR 0027 records the
contract. This change adds one view-replacement migration and one partial
live-body progress index.
- Active branch `fedster99/fix-body-progress-rounding` makes 100% an exact
completion claim for live and priority account progress and per-folder
current body progress. Migration `0023` floors incomplete ratios while
preserving the integer API fields.
- Active branch `fedster99/fix-reconcile-health-after-repair` separates observed reconcile gaps from unresolved reconcile state. A pass that fully tombstones provider-missing rows or backfills missing-in-DB UIDs now finishes clean and can return the account to `HEALTHY`; the run still records its bounded gap count. Missing-in-DB overflow is detected with a 5,001st sentinel row, remains degraded, and retries on the next full-sync cadence. ADR 0026 records the contract; no migration or public API change is required.
- Local branch `fedster99/smtp-account-lock-v2` is rebased onto current
`origin/main` at `88c025d` and is intentionally uncommitted/unpushed. It closes
Expand Down Expand Up @@ -349,10 +356,9 @@ This is the tracked restart point for future agents. Keep it concise, factual, a

## Next Best Actions

- Review and land the feature-branch PR for
`fedster99/live-body-coverage-policy`. Publish a new immutable public-core
image only after human merge, then update downstream Signal through its
public-core prebuild and re-pin flow.
- Review and land `fedster99/fix-body-progress-rounding`. Publish a new
immutable public-core image only after human merge, then update downstream
Signal through its public-core prebuild and re-pin flow.
- Review and land `fedster99/fix-reconcile-health-after-repair`, publish its immutable image, then re-pin downstream consumers. Confirm ordinary provider delete/move drift records a nonzero gap count without leaving a fully repaired account stuck `DEGRADED`.
- Review and land `fedster99/fix-imap-abort-race`, publish its immutable image, then re-pin downstream consumers and canary the Sent/full-sweep deadline boundary. Confirm there are no further `Already logged out`, `process.uncaughtException`, or Render restart events.
- Keep this file updated at the end of substantial sessions.
Expand Down
25 changes: 22 additions & 3 deletions apps/api/src/__tests__/schema.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,10 @@ const contentExtractBodyStoreMigrationPath = resolve(
process.cwd(),
"supabase/migrations/public/0022_content_extract_body_store.sql"
);
const exactBodyCompletionPercentagesMigrationPath = resolve(
process.cwd(),
"supabase/migrations/public/0023_exact_body_completion_percentages.sql"
);

describe("initial schema", () => {
it("adds a bounded search extract without moving full body payloads into metadata", async () => {
Expand Down Expand Up @@ -145,9 +149,9 @@ describe("initial schema", () => {
const version = await getRequiredPublicSchemaVersion();
const sql = await readPublicMigrations();

expect(version).toBe("0022_content_extract_body_store");
expect(version).toBe("0023_exact_body_completion_percentages");
expect(manifest).toEqual({
schemaVersion: "0022_content_extract_body_store",
schemaVersion: "0023_exact_body_completion_percentages",
migrations: [
{ id: "0001_imap_mirror", file: "0001_imap_mirror.sql" },
{ id: "0002_stuck_degraded_escalation", file: "0002_stuck_degraded_escalation.sql" },
Expand All @@ -170,7 +174,11 @@ describe("initial schema", () => {
{ id: "0019_authored_delivery_evidence", file: "0019_authored_delivery_evidence.sql" },
{ id: "0020_threading_fingerprint_closure", file: "0020_threading_fingerprint_closure.sql" },
{ id: "0021_row_accurate_body_progress", file: "0021_row_accurate_body_progress.sql" },
{ id: "0022_content_extract_body_store", file: "0022_content_extract_body_store.sql" }
{ id: "0022_content_extract_body_store", file: "0022_content_extract_body_store.sql" },
{
id: "0023_exact_body_completion_percentages",
file: "0023_exact_body_completion_percentages.sql"
}
]
});
expect(sql).toContain("CREATE TABLE IF NOT EXISTS public.imap_accounts");
Expand Down Expand Up @@ -202,6 +210,17 @@ describe("initial schema", () => {
expect(sql).toContain("FUNCTION public.imap_search_extract_fts");
});

it("keeps incomplete body percentages below 100", async () => {
const sql = await readFile(exactBodyCompletionPercentagesMigrationPath, "utf8");

expect(sql).toContain("CREATE OR REPLACE VIEW public.imap_account_progress");
expect(sql).toContain("b.priority_bodies_fetched_count >= b.priority_bodies_target_count");
expect(sql).toContain("b.live_bodies_fetched_count >= b.live_bodies_target_count");
expect(sql.match(/ELSE floor\(/g)).toHaveLength(2);
expect(sql).not.toContain("stripe");
expect(sql).not.toContain("tenant");
});

it("indexes delivery fingerprints for bounded threading closure", async () => {
const sql = await readFile(threadingFingerprintClosureMigrationPath, "utf8");

Expand Down
91 changes: 91 additions & 0 deletions apps/api/src/__tests__/sync-engine.integration.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2987,6 +2987,97 @@ integration("sync-engine integration (real Postgres + fixture IMAP)", () => {
bodies_pct: 0
})
]));

await h.pool.query(
`
INSERT INTO public.imap_message_bodies (
message_id, raw_mime, raw_bytes, raw_truncated, body_text
)
SELECT id, NULL, 20, false, 'marker repaired'
FROM public.imap_messages
WHERE account_id = $1
AND folder_path = 'INBOX'
AND uidvalidity = 72001
AND uid = 4
`,
[h.account.id]
);
await h.pool.query(
`
INSERT INTO public.imap_messages (
account_id, folder_path, uidvalidity, uid, internal_date,
subject, body_fetched_at, deleted_in_provider, window_status
)
SELECT
$1,
'INBOX',
72001,
uid,
now(),
'rounding boundary ' || uid,
now(),
false,
'IN_WINDOW'
FROM generate_series(7, 202) AS uid
`,
[h.account.id]
);
await h.pool.query(
`
INSERT INTO public.imap_message_bodies (
message_id, raw_mime, raw_bytes, raw_truncated, body_text
)
SELECT id, NULL, 20, false, 'rounding boundary complete'
FROM public.imap_messages
WHERE account_id = $1
AND folder_path = 'INBOX'
AND uidvalidity = 72001
AND uid BETWEEN 7 AND 202
`,
[h.account.id]
);

const roundingBoundaryProgress = (
await h.pool.query<{
priority_bodies_fetched_count: number;
priority_bodies_target_count: number;
priority_bodies_complete_pct: number;
live_bodies_fetched_count: number;
live_bodies_target_count: number;
live_bodies_complete_pct: number;
}>(
`
SELECT
priority_bodies_fetched_count,
priority_bodies_target_count,
priority_bodies_complete_pct,
live_bodies_fetched_count,
live_bodies_target_count,
live_bodies_complete_pct
FROM public.imap_account_progress
WHERE account_id = $1
`,
[h.account.id]
)
).rows[0];
expect(roundingBoundaryProgress).toEqual({
priority_bodies_fetched_count: 199,
priority_bodies_target_count: 200,
priority_bodies_complete_pct: 99,
live_bodies_fetched_count: 199,
live_bodies_target_count: 200,
live_bodies_complete_pct: 99
});

const roundingBoundaryDetails = await h.repository.getAccountDetails(h.account.id);
expect(roundingBoundaryDetails?.folders).toEqual(expect.arrayContaining([
expect.objectContaining({
path: "INBOX",
live_bodies_fetched_count: 199,
live_bodies_target_count: 200,
bodies_pct: 99
})
]));
});

it("Scenario S — settings PATCH makes an existing non-priority live body eligible on the next sync", async () => {
Expand Down
7 changes: 6 additions & 1 deletion apps/api/src/repository.ts
Original file line number Diff line number Diff line change
Expand Up @@ -696,7 +696,12 @@ export class MirrorRepository {
END AS headers_pct,
CASE
WHEN COALESCE(b.live_bodies_target_count, 0) > 0
THEN round((b.live_bodies_fetched_count::numeric / b.live_bodies_target_count::numeric) * 100)::int
THEN CASE
WHEN b.live_bodies_fetched_count >= b.live_bodies_target_count THEN 100
ELSE floor(
(b.live_bodies_fetched_count::numeric / b.live_bodies_target_count::numeric) * 100
)::int
END
WHEN f.live_window_target_count IS NOT NULL THEN 100
ELSE 0
END AS bodies_pct,
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
-- A current live body percentage is complete only when its fetched count
-- reaches its target. Floor incomplete ratios so 199/200 cannot round to 100.
CREATE OR REPLACE VIEW public.imap_account_progress
WITH (security_invoker = true)
AS
WITH folder_progress AS (
SELECT
f.account_id,
count(*) FILTER (WHERE f.live_window_target_count IS NOT NULL)::int AS live_window_known_folder_count,
count(*) FILTER (WHERE f.historical_target_count IS NOT NULL)::int AS historical_known_folder_count,
COALESCE(sum(LEAST(f.headers_synced_count, COALESCE(f.live_window_target_count, 0))), 0)::int
AS live_headers_synced_count,
COALESCE(sum(COALESCE(f.live_window_target_count, 0)), 0)::int AS live_headers_target_count,
COALESCE(sum(GREATEST(f.headers_synced_count - COALESCE(f.live_window_target_count, 0), 0)), 0)::int
AS historical_headers_synced_count,
COALESCE(sum(COALESCE(f.historical_target_count, 0)), 0)::int AS historical_headers_target_count,
COALESCE(sum(GREATEST(f.bodies_fetched_count - COALESCE(f.live_window_target_count, 0), 0)), 0)::int
AS historical_bodies_fetched_count,
COALESCE(sum(COALESCE(f.historical_target_count, 0)), 0)::int AS historical_bodies_target_count
FROM public.imap_folders f
WHERE f.tracked = true
AND f.status != 'MISSING'
GROUP BY f.account_id
),
active_body_folder_progress AS (
SELECT
f.account_id,
count(*) FILTER (WHERE f.live_window_target_count IS NOT NULL)::int AS live_window_known_folder_count,
count(*) FILTER (
WHERE f.sync_priority <= 10
AND f.live_window_target_count IS NOT NULL
)::int AS priority_live_window_known_folder_count
FROM public.imap_folders f
WHERE f.tracked = true
AND f.missing_since IS NULL
AND f.status NOT IN ('MISSING', 'PENDING_VERIFICATION')
GROUP BY f.account_id
),
current_live_body_progress AS (
SELECT
m.account_id,
count(*)::int AS live_bodies_target_count,
count(*) FILTER (
WHERE m.body_fetched_at IS NOT NULL
AND b.message_id IS NOT NULL
AND NOT b.raw_truncated
)::int AS live_bodies_fetched_count,
count(*) FILTER (
WHERE f.sync_priority <= 10
)::int AS priority_bodies_target_count,
count(*) FILTER (
WHERE f.sync_priority <= 10
AND m.body_fetched_at IS NOT NULL
AND b.message_id IS NOT NULL
AND NOT b.raw_truncated
)::int AS priority_bodies_fetched_count
FROM public.imap_messages m
JOIN public.imap_folders f
ON f.account_id = m.account_id
AND f.path = m.folder_path
LEFT JOIN public.imap_message_bodies b
ON b.message_id = m.id
WHERE f.tracked = true
AND f.missing_since IS NULL
AND f.status NOT IN ('MISSING', 'PENDING_VERIFICATION')
AND m.deleted_in_provider = false
AND m.window_status = 'IN_WINDOW'
GROUP BY m.account_id
)
SELECT
a.id AS account_id,
COALESCE(p.live_headers_synced_count, 0) AS live_headers_synced_count,
COALESCE(p.live_headers_target_count, 0) AS live_headers_target_count,
CASE
WHEN COALESCE(p.live_headers_target_count, 0) > 0
THEN LEAST(100, round((p.live_headers_synced_count::numeric / p.live_headers_target_count::numeric) * 100)::int)
WHEN COALESCE(p.live_window_known_folder_count, 0) > 0 THEN 100
ELSE 0
END AS live_headers_complete_pct,
COALESCE(b.priority_bodies_fetched_count, 0) AS priority_bodies_fetched_count,
COALESCE(b.priority_bodies_target_count, 0) AS priority_bodies_target_count,
CASE
WHEN COALESCE(b.priority_bodies_target_count, 0) > 0
THEN CASE
WHEN b.priority_bodies_fetched_count >= b.priority_bodies_target_count THEN 100
ELSE floor(
(b.priority_bodies_fetched_count::numeric / b.priority_bodies_target_count::numeric) * 100
)::int
END
WHEN COALESCE(ab.priority_live_window_known_folder_count, 0) > 0 THEN 100
ELSE 0
END AS priority_bodies_complete_pct,
COALESCE(b.live_bodies_fetched_count, 0) AS live_bodies_fetched_count,
COALESCE(b.live_bodies_target_count, 0) AS live_bodies_target_count,
CASE
WHEN COALESCE(b.live_bodies_target_count, 0) > 0
THEN CASE
WHEN b.live_bodies_fetched_count >= b.live_bodies_target_count THEN 100
ELSE floor(
(b.live_bodies_fetched_count::numeric / b.live_bodies_target_count::numeric) * 100
)::int
END
WHEN COALESCE(ab.live_window_known_folder_count, 0) > 0 THEN 100
ELSE 0
END AS live_bodies_complete_pct,
COALESCE(p.historical_headers_synced_count, 0) AS historical_headers_synced_count,
COALESCE(p.historical_headers_target_count, 0) AS historical_headers_target_count,
CASE
WHEN COALESCE(p.historical_headers_target_count, 0) > 0
THEN LEAST(100, round((p.historical_headers_synced_count::numeric / p.historical_headers_target_count::numeric) * 100)::int)
WHEN COALESCE(p.historical_known_folder_count, 0) > 0 THEN 100
ELSE 0
END AS historical_headers_complete_pct,
COALESCE(p.historical_bodies_fetched_count, 0) AS historical_bodies_fetched_count,
COALESCE(p.historical_bodies_target_count, 0) AS historical_bodies_target_count,
CASE
WHEN COALESCE(p.historical_bodies_target_count, 0) > 0
THEN LEAST(100, round((p.historical_bodies_fetched_count::numeric / p.historical_bodies_target_count::numeric) * 100)::int)
WHEN COALESCE(p.historical_known_folder_count, 0) > 0 THEN 100
ELSE 0
END AS historical_bodies_complete_pct,
NULL::timestamptz AS estimated_full_sync_at
FROM public.imap_accounts a
LEFT JOIN folder_progress p ON p.account_id = a.id
LEFT JOIN active_body_folder_progress ab ON ab.account_id = a.id
LEFT JOIN current_live_body_progress b ON b.account_id = a.id;
6 changes: 5 additions & 1 deletion apps/api/supabase/migrations/public/manifest.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"schemaVersion": "0022_content_extract_body_store",
"schemaVersion": "0023_exact_body_completion_percentages",
"migrations": [
{
"id": "0001_imap_mirror",
Expand Down Expand Up @@ -88,6 +88,10 @@
{
"id": "0022_content_extract_body_store",
"file": "0022_content_extract_body_store.sql"
},
{
"id": "0023_exact_body_completion_percentages",
"file": "0023_exact_body_completion_percentages.sql"
}
]
}
2 changes: 1 addition & 1 deletion docs/agent/reliability-invariants.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ This is the agent-readable reliability contract distilled from `docs/spec-confor
- `live_window_days` is immutable after account creation in v0.1; changing it requires a future window-status migration story.
- `PATCH /accounts/:id/settings` may change `bodyFetchPolicy` only to `immediate`, `lazy`, or `priority_then_backfill`. The endpoint stays strict and rejects invalid, empty, or unknown input.
- `immediate` includes every active live-window message in the automatic body lane. `lazy` disables automatic backlog fetch. `priority_then_backfill` includes only priority folders and does not promise later live-body coverage for current non-priority folders.
- `imap_account_progress` is a read model. Live and priority body fields come from current message/body rows; header and historical fields still use cumulative folder counters. Per-folder `live_bodies_fetched_count`, `live_bodies_target_count`, and `bodies_pct` use current nondeleted `IN_WINDOW` rows and store-completed, non-truncated body evidence for every returned folder. Because the folder list includes inactive folders, its targets need not sum to the active account target. Migration `0021_row_accurate_body_progress` adds `imap_messages_live_body_progress_idx` on `(account_id, folder_path, id)` for active `IN_WINDOW` rows; migration `0022_content_extract_body_store` additionally requires `body_fetched_at` so a pre-store evidence row cannot count as complete. Large existing mirrors must prebuild that exact index concurrently before applying the transactional migration.
- `imap_account_progress` is a read model. Live and priority body fields come from current message/body rows; header and historical fields still use cumulative folder counters. Per-folder `live_bodies_fetched_count`, `live_bodies_target_count`, and `bodies_pct` use current nondeleted `IN_WINDOW` rows and store-completed, non-truncated body evidence for every returned folder. Live, priority, and per-folder current body percentages can equal 100 only when the fetched count reaches the target; incomplete ratios are floored. Because the folder list includes inactive folders, its targets need not sum to the active account target. Migration `0021_row_accurate_body_progress` adds `imap_messages_live_body_progress_idx` on `(account_id, folder_path, id)` for active `IN_WINDOW` rows; migration `0022_content_extract_body_store` additionally requires `body_fetched_at` so a pre-store evidence row cannot count as complete. Large existing mirrors must prebuild that exact index concurrently before applying the transactional migration.
- Account sync runs as three ordered lanes under one advisory lock: hot metadata/reconcile, capped live body backlog, then history.
- History lane work must never run before hot sync or the live body lane, and it must stop when the cooperative lock budget is exhausted.
- Historical backfill uses the folder `backfill_*` state and `last_archive_refresh_at`; it snapshots older-than-window UIDs and walks them newest-first in resumable batches.
Expand Down
Loading