feat(export): Sheet 3 List of Expenses feeding Sheet 2 actuals

openspec/changes/budget-feat-313-excel-export/design.md

@@ -33,6 +33,7 @@ That function sums conversions/expenses per *budget*, across all budgets a custo
 **2. Converted-expense figure blends real allocation rates with `estimated_exchange_rate` for the unsatisfied remainder, computed per report line then summed per budget line.**
 For each report line: sum `allocation.amount_allocated × (conversion.donor_amount ÷ conversion.local_amount)` across its allocations (real rate, since `amount_allocated` is in `local_currency`), then add `(report_line.amount − Σallocation.amount_allocated) ÷ estimated_exchange_rate` for any remainder. This mirrors the existing precedent of using `estimated_exchange_rate` as an approximate stand-in (donor dashboard, budget-line toggle) rather than inventing a new conversion rule. A budget line is flagged "includes estimate" in the export if any of its report lines have a non-zero unsatisfied remainder.
 *Alternative considered*: omit the unsatisfied remainder entirely (leave it unconverted/blank). Rejected per explicit product decision — donors expect the dashboard total to foot to something close to the full spend, and GrandFlow already accepts approximate figures elsewhere rather than showing gaps.
+*Superseded by Decision 14*: the blend still happens, but no longer as a Python rollup — Sheet 3 now carries the estimated-rate fallback per row, and Sheet 2 sums it via an Excel `SUMIF` instead of `get_budget_line_expense_rollups`/`_compute_converted_expense` (both removed).
 
 **3. Workbook generated synchronously in the request/response cycle via `openpyxl.Workbook()`, streamed with `StreamingResponse` (mirroring `attachment_routes.py`), not written to storage first.**
 A single budget's data (lines, ledger, report lines) is small; no need for the async/background pattern used elsewhere (e.g. Celery) for larger jobs.
@@ -76,6 +77,11 @@ Donors don't reason in local currency (per user feedback), so every comparison f
 *Deferred*: the reference file's "Avg rate" idea (a single blended rate to normalize the Funding Ledger and replace its per-row literals with formulas) is explicitly punted to group 3 (List of Expenses), per the user: "leave it for now, we will come back to this, whenever we build list of expenses page."
 *Verification*: the implementation was diff-checked cell-by-cell against the reference file (matching except for the user's own placeholder/typo cells and float-precision artifacts from hand-typing) and round-tripped through `soffice --headless` to confirm the formulas compute, not just parse — see Decision 5's same verification approach.
 
+**14. Sheet 3's unallocated remainder uses `estimated_exchange_rate` (flagged), and Sheet 2's expense/converted-expense columns are `SUMIF` formulas reading Sheet 3, not Python rollups.**
+Revised after real-data testing on group 3 surfaced two problems: an expense with zero allocations showed no footnote explaining its blank cells, and Sheet 2 highlighted a cell with no visible reason why. Root cause of the second: Sheet 2 and Sheet 3 computed the same blended figure two different ways (a Python rollup vs. per-allocation rows), so Sheet 3 could show real numbers next to a blank rate while Sheet 2 silently blended in an estimate. Fix: Sheet 3's remainder row (design.md's "not yet allocated" case) now uses `budget.estimated_exchange_rate` as its rate when the budget has one, same convention as Sheet 1/2 (`local ÷ rate = donor`), flagged with a red `*` (row) and a footnote explaining the estimate; when the budget has no `estimated_exchange_rate` at all, the row still has no rate (no fabrication from nothing). Sheet 2 then reads Sheet 3 directly — each budget line's row carries a hidden `budget_line_id` (Sheet 2 column H, Sheet 3 column I) and its Expenses/Total-Expenses cells are `=SUMIF('List of Expenses'!$I:$I,$H<row>,'List of Expenses'!$E:$E resp. $F:$F)` — instead of `get_budget_line_expense_rollups`/`_compute_converted_expense` (both removed from `excel_export_crud.py`/`excel_export_service.py`). Matching by a hidden id column, not the visible description text, avoids two same-named budget lines silently double-counting. `_build_expense_rows` (module-level, was `ExpenseListSheet._build_rows`) now builds this row list once per export and feeds both sheets, so they can no longer disagree. A `Category` column was also added to Sheet 3 (after Date), since the sheet otherwise has no way to tell two identically-named budget lines in different categories apart when scanning it directly.
+*Alternative considered*: keep Sheet 3's remainder blank and have Sheet 2 keep its own Python blend. Rejected — that's the two-sources-of-truth setup that caused the bug; a single computed row list removes the possibility of the sheets disagreeing.
+*Verification*: round-tripped through `soffice --headless`, confirming the `SUMIF` formulas compute the same blended total the old Python rollup did.
+
 ## Risks / Trade-offs
 
 - [A budget with many report lines/allocations could make generation slow] → Out of scope for a "simple export" of one budget; single-budget expense volume in practice is small (tens to low hundreds of lines), revisit only if real usage shows otherwise.

openspec/changes/budget-feat-313-excel-export/specs/budget-excel-export/spec.md

@@ -63,7 +63,7 @@ For each budget line (and category subtotal), Sheet 2 SHALL show: total expenses
 - **THEN** its expense and converted-expense figures show zero, and its deviation equals its full budgeted amount
 
 ### Requirement: Sheet 3 — List of Expenses with per-allocation sublines
-The workbook's third sheet SHALL list every report-line expense across all of the budget's reports, one row per report line when its full amount is covered by a single currency-conversion lot, or one row per allocation (subline) when a report line's amount is funded by more than one lot — each subline row carrying that allocation's own conversion date, converted amount, and implied rate.
+The workbook's third sheet SHALL list every report-line expense across all of the budget's reports, one row per report line when its full amount is covered by a single currency-conversion lot, or one row per allocation (subline) when a report line's amount is funded by more than one lot — each subline row carrying that allocation's own conversion date, converted amount, and implied rate. Each row SHALL also show the expense's budget-line category.
 
 #### Scenario: Expense funded by a single lot
 - **WHEN** a report-line expense is fully allocated to exactly one currency-conversion lot
@@ -73,9 +73,13 @@ The workbook's third sheet SHALL list every report-line expense across all of th
 - **WHEN** a report-line expense straddles more than one currency-conversion lot (e.g., one receipt converted across four separate events)
 - **THEN** Sheet 3 shows one row per allocation, each with its own conversion date and implied rate, and the rows' amounts sum to the expense's full amount
 
-#### Scenario: Expense not yet allocated
-- **WHEN** a report-line expense has no currency-conversion allocation at all
-- **THEN** Sheet 3 shows it as a single row with its local-currency amount and no conversion date or rate
+#### Scenario: Expense not yet allocated, budget has an estimated rate
+- **WHEN** a report-line expense has no currency-conversion allocation at all, and the budget has an `estimated_exchange_rate`
+- **THEN** Sheet 3 shows it as a single row with its local-currency amount, no conversion date, and the budget's estimated rate — flagged (a marker in place of the conversion date, and a footnote) as using the estimate rather than a recorded conversion
+
+#### Scenario: Expense not yet allocated, no estimated rate available
+- **WHEN** a report-line expense has no currency-conversion allocation at all, and the budget has no `estimated_exchange_rate`
+- **THEN** Sheet 3 shows it as a single row with its local-currency amount and no conversion date, rate, or converted amount, flagged with a footnote explaining the omission
 
 ### Requirement: Available export templates for a budget
 The system SHALL provide an endpoint listing the export templates a given viewer may use for a given budget: the system default, every template owned by the viewer's organisation, and every template owned by the budget's funder that is marked shared with grantees. Each entry SHALL identify whether it is the system default, the viewer's own, or a funder's.

openspec/changes/budget-feat-313-excel-export/tasks.md

@@ -35,12 +35,12 @@ _Generated by `scripts/flow.py sync budget-feat-313-excel-export` — do not edi
 - [x] 2.3 Implement the converted-expense calculation in `excel_export_service.py`: real per-allocation rate for allocated amounts plus `estimated_exchange_rate` for any unsatisfied remainder, flagging a line as "includes estimate" when a remainder exists; verify with a unit test covering fully-allocated (no flag), partially-allocated (flagged, blended figure), and fully-unallocated (fully estimated, flagged) cases
 - [x] 2.4 Implement Sheet 2 as a `DashboardSheet(_SheetWriter)` subclass: an income section (one row per `CurrencyConversion` for the budget — converted date, donor amount, local amount, implied rate — plus a total row); verify with a unit test using a budget with one funding receipt and multiple conversions, asserting one row per conversion — **superseded**: revised into the Funding Ledger/Report Summary/donor-currency-first design in Decision 13 after a user-supplied reference file; the income-section shape described here is not what shipped
 - [x] 2.5 Add `DashboardSheet`'s per-budget-line and category-subtotal rows (expenses local, expenses converted, deviation), reusing Sheet 1's category grouping/order and applying the estimated-portion cell style (italic + fill, per design.md Decision 4) where flagged; verify with a unit test asserting column values and that flagged cells carry the style — **superseded**: see Decision 13; deviation is donor-currency (Original − Total Expenses), not local-currency
-- [ ] 2.6 Run backend lint/tests clean for `services/budget`; PR merged (`Closes` this group's sub-issue)
+- [x] 2.6 Run backend lint/tests clean for `services/budget`; PR merged (`Closes` this group's sub-issue)
 
 ## 3. Sheet 3 — List of Expenses with per-allocation sublines — depends on 1 — Issue #321
 
-- [ ] 3.1 Extend `excel_export_crud.py` with a query returning every report line across all of the budget's reports, each with its ordered list of allocations (or none)
-- [ ] 3.2 Implement Sheet 3 as an `ExpenseListSheet(_SheetWriter)` subclass (see design.md Decision 12): one row per report line when zero or one allocation exists, one row per allocation (subline) when a report line has more than one, each subline carrying its own conversion date/rate/converted amount; verify with a unit test covering an unallocated expense, a single-lot expense, and a multi-lot expense (asserting the multi-lot rows sum to the expense's full amount)
+- [x] 3.1 Extend `excel_export_crud.py` with a query returning every report line across all of the budget's reports, each with its ordered list of allocations (or none)
+- [x] 3.2 Implement Sheet 3 as an `ExpenseListSheet(_SheetWriter)` subclass (see design.md Decision 12): one row per report line when zero or one allocation exists, one row per allocation (subline) when a report line has more than one, each subline carrying its own conversion date/rate/converted amount; verify with a unit test covering an unallocated expense, a single-lot expense, and a multi-lot expense (asserting the multi-lot rows sum to the expense's full amount) — **extended** (design.md Decision 14, after real-data testing): added a Category column; an unallocated remainder now uses `estimated_exchange_rate` (flagged) instead of being left blank when the budget has one; Sheet 2's expense columns now `SUMIF` this sheet via a hidden budget-line-id column instead of a separate Python rollup
 - [ ] 3.3 Run backend lint/tests clean for `services/budget`; PR merged (`Closes` this group's sub-issue)
 
 ## 4. Frontend export button — depends on 1, 2, 3 — Issue #322

scripts/git-hooks/pre-push

@@ -65,9 +65,13 @@ for area in "${!areas[@]}"; do
   case "$area" in
     ai|budget|chat|users)
       echo "== services/$area =="
-      run black --check "services/$area/app"
-      run mypy "services/$area/app"
-      run flake8 "services/$area/app"
+      # Run from the service dir like CI: from the repo root, app.* imports don't
+      # resolve and mypy.ini's ignore_missing_imports silently turns them into Any.
+      pushd "services/$area" > /dev/null
+      run black --check ./app
+      run mypy ./app
+      run flake8 ./app
+      popd > /dev/null
       ;;
     worker)
       echo "== services/worker =="

services/budget/app/crud/excel_export_crud.py

@@ -1,80 +1,95 @@
+from datetime import date
 from dataclasses import dataclass, field
 from uuid import UUID
 
-from sqlalchemy import func, select
+from sqlalchemy import select
 from sqlalchemy.ext.asyncio import AsyncSession
 
-from app.models.budget import BudgetLineModel
 from app.models.currency_ledger import CurrencyConversionModel, ReportLineConversionAllocationModel
-from app.models.report import ReportLineModel
+from app.models.report import ReportLineModel, ReportModel
 
 
 @dataclass
-class AllocationRollup:
-    """One report line's expense funded by one currency-conversion lot."""
+class ReportLineAllocationDetail:
+    """One allocation (subline) of a report-line expense: its own conversion
+    date/amounts, oldest-first (FIFO consumption order)."""
 
     amount_allocated: float
+    converted_at: date
     conversion_donor_amount: float
     conversion_local_amount: float
 
 
 @dataclass
-class BudgetLineExpenseRollup:
-    """A budget line's total local-currency expenses plus funding allocations."""
+class ReportLineExpense:
+    """One report-line expense with its ordered currency-conversion allocations."""
 
+    report_line_id: UUID
     budget_line_id: UUID
-    total_local_amount: float
-    allocations: list[AllocationRollup] = field(default_factory=list)
+    description: str | None
+    amount: float
+    expense_date: date
+    allocations: list[ReportLineAllocationDetail] = field(default_factory=list)
 
 
-async def get_budget_line_expense_rollups(
+async def get_report_line_expenses(
     session: AsyncSession, budget_id: UUID
-) -> dict[UUID, BudgetLineExpenseRollup]:
-    """One rollup per budget line of `budget_id`, keyed by budget_line_id."""
-    totals_sq = (
+) -> list[ReportLineExpense]:
+    """Every report-line expense across all of `budget_id`'s reports, oldest
+    expense first, each with its ordered (oldest-first) allocations."""
+    lines_result = await session.execute(
         select(
-            ReportLineModel.budget_line_id.label("budget_line_id"),
-            func.sum(ReportLineModel.amount).label("total_local_amount"),
+            ReportLineModel.id,
+            ReportLineModel.budget_line_id,
+            ReportLineModel.description,
+            ReportLineModel.amount,
+            ReportLineModel.expense_date,
         )
-        .group_by(ReportLineModel.budget_line_id)
-        .subquery()
-    )
-    totals_result = await session.execute(
-        select(BudgetLineModel.id, func.coalesce(totals_sq.c.total_local_amount, 0.0))
-        .outerjoin(totals_sq, totals_sq.c.budget_line_id == BudgetLineModel.id)
-        .where(BudgetLineModel.budget_id == budget_id)
+        .join(ReportModel, ReportModel.id == ReportLineModel.report_id)
+        .where(ReportModel.budget_id == budget_id)
+        .order_by(ReportLineModel.expense_date, ReportLineModel.id)
     )
-    rollups = {
-        line_id: BudgetLineExpenseRollup(budget_line_id=line_id, total_local_amount=total)
-        for line_id, total in totals_result.all()
-    }
-    if not rollups:
-        return rollups
+    expenses: dict[UUID, ReportLineExpense] = {}
+    for report_line_id, budget_line_id, description, amount, expense_date in lines_result.all():
+        expenses[report_line_id] = ReportLineExpense(
+            report_line_id=report_line_id,
+            budget_line_id=budget_line_id,
+            description=description,
+            amount=amount or 0.0,
+            expense_date=expense_date,
+        )
+    if not expenses:
+        return []
 
     allocations_result = await session.execute(
         select(
-            ReportLineModel.budget_line_id,
+            ReportLineConversionAllocationModel.report_line_id,
             ReportLineConversionAllocationModel.amount_allocated,
+            CurrencyConversionModel.converted_at,
             CurrencyConversionModel.donor_amount,
             CurrencyConversionModel.local_amount,
         )
-        .join(
-            ReportLineModel,
-            ReportLineModel.id == ReportLineConversionAllocationModel.report_line_id,
-        )
         .join(
             CurrencyConversionModel,
             CurrencyConversionModel.id == ReportLineConversionAllocationModel.conversion_id,
         )
-        .where(ReportLineModel.budget_line_id.in_(rollups.keys()))
+        .where(ReportLineConversionAllocationModel.report_line_id.in_(expenses.keys()))
+        .order_by(CurrencyConversionModel.converted_at, CurrencyConversionModel.id)
     )
-    for budget_line_id, amount_allocated, donor_amount, local_amount in allocations_result.all():
-        rollups[budget_line_id].allocations.append(
-            AllocationRollup(
+    for (
+        report_line_id,
+        amount_allocated,
+        converted_at,
+        donor_amount,
+        local_amount,
+    ) in allocations_result.all():
+        expenses[report_line_id].allocations.append(
+            ReportLineAllocationDetail(
                 amount_allocated=amount_allocated,
+                converted_at=converted_at,
                 conversion_donor_amount=donor_amount,
                 conversion_local_amount=local_amount,
             )
         )
 
-    return rollups
+    return list(expenses.values())