Skip to content

perf: O(n²) transaction matching in sync causes slowdown with large datasets #200

@moltboie

Description

@moltboie

Problem

getRemovedTransactions and getRemovedInvestmentTransactions in sync-simple-fin.ts use .find() inside a .forEach() loop, making them O(n²):

// sync-simple-fin.ts
storedTransactions.forEach((t) => {
  const found = transactions.find((f) => f.transaction_id === transaction_id);
  if (!found) removedTransactions.push({ transaction_id });
});

Similarly, sync-plaid.ts has the same pattern in modelize:

const existing = storedTransactions.find((f) => {
  const idMatches = [f.transaction_id, f.pending_transaction_id].includes(e.transaction_id);
  const accountMatches = e.account_id === f.account_id;
  // ...
});

With hundreds of transactions per sync window, this is fine. But as account history grows (2-year lookback) or with many accounts, this becomes a bottleneck.

Fix

Pre-build lookup Maps/Sets before the loop:

const transactionIds = new Set(transactions.map(t => t.transaction_id));
storedTransactions.forEach((t) => {
  if (!transactionIds.has(t.transaction_id)) {
    removedTransactions.push({ transaction_id: t.transaction_id });
  }
});

For the Plaid modelize case, build a composite Map keyed by transaction_id and pending_transaction_id.

Affected files

  • src/server/lib/compute-tools/sync-simple-fin.ts (getRemovedTransactions, getRemovedInvestmentTransactions)
  • src/server/lib/compute-tools/sync-plaid.ts (modelize)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    Status

    Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions