diff --git a/AGENTS.md b/AGENTS.md index 00a7824..d6e1fac 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -35,9 +35,9 @@ propagation in its PR. ## Durable work state -- OpenSpec change `clarify-operator-actions` is planning-complete and validated. Planning PRs 319 and 320 are merged. -- Beads parent `infra-8tt.93` remains in progress. Implementation child `infra-8tt.93.1` is open and owns the ten-task checklist in `openspec/changes/clarify-operator-actions/tasks.md`. -- No runtime implementation or release child exists yet. Create the release child only after offline implementation acceptance. +- OpenSpec change `clarify-operator-actions` is implemented and archived at `openspec/changes/archive/2026-10-27-clarify-operator-actions/`. All ten tasks have offline acceptance proof. +- Beads implementation `infra-8tt.93.1` is complete. Release child `infra-8tt.93.2` owns publication, exact-candidate lab qualification and stable promotion. +- Parent `infra-8tt.93` remains in progress until release acceptance. No publication or production GO was granted by implementation. ## Working mode: fast-track diff --git a/README.md b/README.md index c6a9683..76ac5d7 100644 --- a/README.md +++ b/README.md @@ -139,6 +139,18 @@ Set the required credentials and target details in `.env`. The [full configurati Secrets belong in the environment or a private `.env` file with restricted permissions. Do not place them in Compose committed to Git. Use `docker compose config --quiet` for validation; resolved output can expose secrets. +## Operational results + +Required user actions send one consolidated notification per completed run, +including simulation, until current evidence resolves them. Clean success sends +no heartbeat. Private logs contain the affected scope and safe next action. +Notification delivery does not change financial processing or exit codes. + +`Cycle result` separates success, success with user action required, and failure. +Per-account imports and cash are separate evidence. `request reported successful` +for cash does not claim independent balance readback or release activity holds. +Unavailable evidence is shown explicitly. See [configuration](docs/configuration.md). + ## Corporate-action safety The Flex query must select **Corporate Actions**, including when it is empty. diff --git a/docs/activity-review.md b/docs/activity-review.md index 804e731..7a6ad10 100644 --- a/docs/activity-review.md +++ b/docs/activity-review.md @@ -188,3 +188,27 @@ Offline journal inspection can prepare a guarded reverse payload for an unchange confirmed row only. Reversal requires separate authority and never runs automatically. See [maintenance](cleanup.md#activity-review-journals) and [release qualification](releasing.md#activity-review-qualification). + + +### User action required and container path + +The private log names the configured ACTIVITY_REVIEW_FILE container path, section, +proposal key, held event and reason. A proposal count is not a held-event count. +Resolve the host path from your Compose volume mapping. For example, +`./review:/state` maps `/state/activity-review.yaml` to +`./review/activity-review.yaml`, relative to the Compose project directory. +The connector does not discover host mounts. + +For an eligible `normal run` proposal, compare current and proposed values and +execution path. Edit only `approve` between runs. Only lowercase literal `yes` +approves that exact eligible proposal. Never approve proposals in bulk. Simulation +executes no financial write. Already approved work waiting for the qualified limit +remains automatic work; do not approve again or force reruns. + +For missing proof, invalid/stale choices or conflicts, inspect the reason and +restore current source and destination proof. `yes` cannot make blocked work +executable. Historical maintenance needs the documented preparation and separate +operation authorization; normal sync cannot execute it. For an uncertain request, +inspect the preserved journal and exact destination readback. Do not resend, +delete or edit the journal to release a hold. False matches need the documented +manual resolution and retain separate cleanup authority. diff --git a/docs/configuration.md b/docs/configuration.md index ec2c47e..e89995a 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -257,10 +257,9 @@ Set `DRY_RUN=1` explicitly for the preview. The full pipeline runs (fetch, conve Set a valid IANA `TZ`, such as `Europe/Paris`, even for one-off Docker runs. Missing or invalid `TZ` forces every account into `DRY_RUN` before processing, with one WARNING explaining the reason. No activities, cash or platform are -written. With explicit false `DRY_RUN`, the final notification includes the -reason when Apprise is configured, even if all accounts succeed. Successful -simulations selected by default, blank or truthy `DRY_RUN` send no notification, -including expiry-only reminders and previews. Failed simulations send one +written. The final notification includes the reason when Apprise is configured, +even in default, blank or truthy simulation. Required token renewal also notifies +in simulation. A deliberate clean simulation alone sends no notification. Failed simulations send one configured Failure notification. After review, set `DRY_RUN=false` and retain valid `TZ` to permit writes. @@ -306,7 +305,8 @@ withholding; signed withholding and refund rows and their total remain separate evidence. Coherence requires exact quantity, gross and fee agreement; a complete representation with a discrepancy is not coherent. The diagnostic adds no reads, writes or files. Findings do not change -import selection, cash updates, exit codes or notifications. Live runs do not +import selection, cash updates or exit codes. Required user checks join the +consolidated notification. Live runs do not run this comparison. Use the one-off command below without `CRON`. `absent` means no canonical destination comment was found. It does not establish @@ -316,8 +316,8 @@ conflicting security/date context and missing/conflicting accruals stay uncertai Missing destination profile ISIN is unavailable evidence and permits amount comparison; a nonempty conflicting ISIN blocks identity. Details name the active, activity-ID, data-source, symbol, local-date and ISIN checks with -expected and observed values. INFO counts failed checks and shows anomaly -blocks with securities, source and destination UTC/local dates, activity IDs, +expected and observed values. INFO counts failed checks. WARNING shows actionable +anomaly blocks with securities, source and destination UTC/local dates, activity IDs, quantity, gross, fee and currency. An `absent` group suppressed by deduplication shows the skip path that import selection takes: `skipped by historical comment` when the account has an old `dividend##` comment (active or not), @@ -333,13 +333,13 @@ verify older history. No category authorizes a repair. Use a one-off container with no `CRON` environment variable. The command reads only the report generated by its configured Flex query; Docker does not extend -the broker's report window. INFO includes anomaly and uncertainty details, IDs +the broker's report window. WARNING includes actionable anomaly and uncertainty details, IDs and source references; DEBUG also includes coherent comparisons. Keep Docker logs private and capture them before removing a container if you need evidence. The diagnostic creates no persistent report and log retention depends on Docker's logging configuration. Financial state remains unchanged, but authentication and IBKR report generation still contact the configured services. Technical read -failures still fail the run; diagnostic findings do not send notifications. +failures still fail the run; actionable diagnostic findings join one notification. ```bash docker run --rm --network your-ghostfolio-network -e DRY_RUN=1 -e LOG_LEVEL=DEBUG \ @@ -351,7 +351,7 @@ docker run --rm --network your-ghostfolio-network -e DRY_RUN=1 -e LOG_LEVEL=DEBU ghcr.io/flowcool/ghostfolio-ibkr-sync:latest ``` -The dividend comparison ends with one line, `[DRY RUN] Dividend report: N to check, N informational, N noise`, followed by the detail of each item to check. Findings are written in plain English: the reason, the IBKR and Ghostfolio values side by side, one action line and the statement that nothing was written. Coherent and outside-window findings appear at `DEBUG` only. Differences of less than half a minor unit are noise; a difference of exactly one minor unit with equal quantity is named "minimal difference, cause unknown"; a legacy-comment row on the same security and day is named "probable duplicate". Each item to check that has a candidate carries a "Proposed match": the IBKR dividend and the candidate Ghostfolio rows, each labelled Keep, Duplicate or Differs, with a "strong" or "weak" confidence and the row to act on; with no candidate it reads "no confident match". A proposal never writes, repairs or changes the counts, exit code or notifications. +The dividend comparison ends with one line, `[DRY RUN] Dividend report: N to check, N informational, N noise`, followed by the detail of each item to check. Findings are written in plain English: the reason, the IBKR and Ghostfolio values side by side, one action line and the statement that nothing was written. Coherent and outside-window findings appear at `DEBUG` only. Differences of less than half a minor unit are noise; a difference of exactly one minor unit with equal quantity is named "minimal difference, cause unknown"; a legacy-comment row on the same security and day is named "probable duplicate". Each item to check that has a candidate carries a "Proposed match": the IBKR dividend and the candidate Ghostfolio rows, each labelled Keep, Duplicate or Differs, with a "strong" or "weak" confidence and the row to act on; with no candidate it reads "no confident match". A proposal never writes, repairs or changes the counts or exit code. Formatting does not add a second notification. Look for lines starting with `[DRY RUN]`; when there is nothing to import the log simply says `No new activities to import`. Replace `your-ghostfolio-network` with the Docker network shared with Ghostfolio. Run it again after upgrading to a new version: comparing the log with the previous version's shows exactly what the new version changes. @@ -382,12 +382,12 @@ No ENV change is needed. This is connector policy based on IBKR's [Version 3 error table](https://www.ibkrguides.com/clientportal/performanceandstatements/flex3error.htm), not a guarantee that IBKR recovers. -## Failure notifications (Apprise) +## Notifications (Apprise) Optional. When `APPRISE_URLS` is set, a run that **completes with exit 1** sends one notification through [Apprise](https://github.com/caronc/apprise/wiki), which supports ntfy, Gotify, Telegram, Discord, Slack, email and many more. A completed successful run -in explicit live mode also sends a reminder when its legacy `GHOST_TOKEN` is about to expire (see +in any simulation or live mode also sends a reminder when its legacy `GHOST_TOKEN` is about to expire (see [Token expiry](troubleshooting.md#token-expiry)); the reminder keeps exit 0. Give it a JSON list (up to 10 URLs, each up to 2048 characters; put one URL per entry, as an entry holding several space- or comma-separated URLs counts each one toward the limit of 10): @@ -411,7 +411,7 @@ of days remaining (`0` means expired). If the run also fails, the reminder is in in the same failure notification. At most one notification is sent per completed run; there is no daily state, so multiple runs in one day can repeat the reminder. -With explicit false `DRY_RUN`, missing or invalid IANA `TZ` also sends a final +Missing or invalid IANA `TZ` in any mode also sends a final notification naming the forced-preview reason, even on success. It shares the single summary with any failures or expiry reminder. Notification delivery does not change exit status. @@ -420,12 +420,11 @@ including default, blank or truthy `DRY_RUN` and invalid-value rejection. Failur with an expiry reminder or forced preview uses the same summary, never a second dispatch. Delivery does not change exit status. -**When nothing is sent:** successful default/blank/truthy simulations, including -expiry-only reminders and previews; successful live runs without an expiry -reminder or forced preview; runs with only other warnings (for example unmapped -ISINs); runs stopped by a signal; and when `APPRISE_URLS` is blank or `[]`. If -`APPRISE_URLS` or `APPRISE_TIMEOUT` is invalid, the run logs -`APPRISE_URLS or APPRISE_TIMEOUT is invalid: failure notifications disabled` and syncs as usual. +**When nothing is sent:** clean successful runs without a required action or an +eligible corporate-action notice; deliberate simulation alone; interrupted runs; +and disabled or invalid Apprise settings. Mapping, review, diagnostic checks and +other explicit required actions repeat once per completed run until resolved. +See [User action required](#user-action-required) and [Notification result](#notification-result). **Delivery:** Apprise runs in a separate short-lived worker process with only network settings in its environment (`PATH`, locale, CA bundle and proxy variables; never the IBKR or @@ -505,3 +504,87 @@ without a compatibility launcher, so update any script or note that calls it. - **Active position context** - drafts and excluded activities/accounts retain their import identifiers for deduplication, but cannot contribute holdings, manual matches or dividend proximity evidence. Holdings and manual reconciliation use active YAHOO profiles; other sources cannot back YAHOO sells. Excluded target accounts are refused. Future timestamps are deferred with exit 1 until a later run; this conservative instant check avoids server-local draft assignment even when import responses omit tags. Accepted imports use the actual server profile symbol; a canonicalized symbol can require an explicit mapping before sells under the original ticker are allowed. - **Batch guard limitations** - the gate checks net quantity, not chronological balances. Buy and sell phases are separate requests, so successful buys can remain if sells fail. A dry-run previews candidates without verifying server acceptance. - **No overlap protection** - avoid concurrent sync runs targeting the same Ghostfolio account; deduplication is based on a snapshot read at startup. + + +### User action required + +Required actions send one consolidated reminder on every completed run, including +simulation. Repeated observations of one event and reason count once. Reminders +continue while current evidence requires action. Clean success sends no heartbeat. +Review approval does not suppress a reminder when missing proof still blocks work. + + +### Notification result + +Each completed run reports `not required`, `disabled`, `invalid configuration`, +`sent`, `failed`, or `timeout, delivery unconfirmed`. Configure APPRISE_URLS and +APPRISE_TIMEOUT when a required notification is disabled or invalid. `sent` +means the transport reported success; it does not prove device receipt. A timeout +can still deliver later. The connector makes one attempt and does not retry. +Delivery failure does not change the financial result or process exit code. + + +Mapping and trade reminders use `mapping_unverified`, `mapping_conflict`, +`trade_manual`, `trade_position` or `trade_invalid`. Private logs identify the +source scope. Verify the intended security in Ghostfolio or correct MAPPING_FILE. +For manual matches, compare source and destination before any reference repair. +Current valid source fields and verified destination ownership prove resolution. +Mapping checks still run for already imported trades in the acquired window. +Normal CASH/OPT exclusions, absent positions and recovered network retries do not +create required-action reminders by themselves. + + +Dividend actions use `dividend_rejected` for missing/conflicting cash, quantity, +payment association or withholding evidence. `dividend_reversal` identifies an +orphan reversal. `dividend_signed_tax` identifies signed zero-net tax, orphan tax +or a withholding refund. `dividend_check` identifies a discrepant proximity skip. +Inspect preserved Flex and Ghostfolio evidence. Never invent payment or quantity, +and never resend an uncertain transaction. Complete consistent current evidence, +verified destination agreement or documented manual resolution proves resolution. +Ordinary zero-net payments without signed tax remain exclusions. Future dates +remain deferred work. A representation limit alone does not request correction. + + +Simulation reports dividend items to check at WARNING. Concrete comparisons +that require intervention join the same `dividend_check` reminder. Inspect the +named activity and both sides of the comparison. Current agreement or documented +manual resolution proves resolution. Information, accepted rounding and noise +alone send no reminder. Named representation or coverage limits remain visible +as evidence limitations without requesting an impossible correction. No diagnostic +changes financial selection, grants cleanup authority or writes a correction. + + +Corporate-action holds and concrete historical discrepancies repeat in the same +summary as other actions. Private conclusions name scope and the configured state +path. Restore the stated evidence; acknowledgment and state reset do not release +holds. Nonblocking findings without a user action keep first/change/resumption +notifications. Automatic evidence maintenance remains a retry and is reported as +such; it does not request approval. Token renewal and invalid or missing TZ also +notify in simulation. Correct the named setting; simulation does not resolve it. + + +### Cycle result + +Completed cycles report `success`, `success with user action required`, or +`failure`. Failure takes priority. Exit codes keep their existing meaning. +Accepted imports or simulated proposals, duplicate skips, unique review-held +events, other holds and deferred automatic work appear separately. Decision +counts can overlap. Unknown evidence reads `unavailable`, never an invented zero. +Cash is reported per account as `simulated`, `request reported successful`, +`not updated` or `uncertain/failed`. Request success is not independent balance +readback. Cash success does not mean held activities were imported. + +For example, `success with user action required` can report zero accepted imports, +review-held events and `cash=request reported successful`. The notification result +appears beside the cycle result. Automatic deferral alone remains `success`. + + +### Application log levels and streams + +DEBUG and INFO use stdout. WARNING and above use stderr. Each application record +has one timestamp and one level. The scheduler passes application output directly +and keeps its lifecycle messages separate. A transport stream is not a severity. +For a clean scheduled run, application INFO stays INFO without scheduler wrapping. +`LOG_LEVEL` still filters application records. LAB_EVIDENCE JSON remains at DEBUG. +Startup shows `ghostfolio-ibkr-sync build `. This identifies the build; +a stable tag can reuse an RC digest and retain the qualified RC build identity. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 5b61b73..547e47c 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -61,7 +61,7 @@ Before any import, the tool reads all existing Ghostfolio activities in one requ A run is best-effort: an error on one account is logged and the remaining accounts are still processed, but any failure makes the whole run exit 1. Failures that count include an IBKR Flex Query fetch error, a Flex query without the time-zone suffix or the required dividend fields, a refused cash balance (currency mismatch or no base currency), an unknown trade side, a Ghostfolio account name that does not exist, an import returning 4xx/5xx, a failed cash balance update, and an unexpected error while processing the account (logged with its traceback; the other accounts are still processed). -With [failure notifications](configuration.md#failure-notifications-apprise) configured, every exit-1 run sends one notification after all accounts were processed, except a dry run chosen with `DRY_RUN`, which never notifies. A run forced into `DRY_RUN` by a missing or invalid `TZ` does notify, even when every account succeeds. Delivery never changes the exit code. +With [notifications](configuration.md#notifications-apprise) configured, every completed exit-1 run sends one failure summary, including simulation. Required user actions join one summary per completed run until current evidence resolves them. Missing or invalid `TZ` also notifies in simulation. Clean success sends no heartbeat. Delivery never changes the exit code. Unmapped ISINs are **not** a failure - they are reported at the end of the run as a prompt to update your mapping file, and the run still exits 0. Activities that Ghostfolio itself detects as duplicates are not a failure either. The tool logs `Ghostfolio created X of Y` and counts only submitted rows matched to the server-created activity list as imported. An unknown or unverifiable import outcome fails the account and blocks subsequent imports and processing of later queries targeting that account for the rest of the run. diff --git a/docs/upgrading-to-2.2.md b/docs/upgrading-to-2.2.md index 057cd4b..4f55845 100644 --- a/docs/upgrading-to-2.2.md +++ b/docs/upgrading-to-2.2.md @@ -8,7 +8,7 @@ additions are worth making: in at every run. Set only one of the two: both together stop the run with exit 1. See [Choose how the sync authenticates](configuration.md#configuration-reference). 2. **Turn on failure alerts** with `APPRISE_URLS` (and optionally `APPRISE_TIMEOUT`). See - [Failure notifications](configuration.md#failure-notifications-apprise). + [Failure notifications](configuration.md#notifications-apprise). Then pull the image and **recreate** the container (a restart keeps the old image), run once with `DRY_RUN=1`, and check the version in the start-up log. The image now also contains Apprise diff --git a/docs/upgrading-to-3.0.md b/docs/upgrading-to-3.0.md index 76bf327..1ca88d4 100644 --- a/docs/upgrading-to-3.0.md +++ b/docs/upgrading-to-3.0.md @@ -63,8 +63,9 @@ the container. The image sets no default. **Symptom if not done.** Nothing is imported, and the log shows `TZ is missing: whole run forced to DRY_RUN, no Ghostfolio writes` (or `TZ is not a valid IANA timezone: ...`). With `APPRISE_URLS` set, one notification names the reason, -even when every account succeeds, if `DRY_RUN` is explicitly false. Successful -default/blank/truthy simulations stay silent; failed simulations send Failure. +even when every account succeeds, including default/blank/truthy simulation. +Required user actions repeat in the same consolidated summary until resolved. +Clean success sends no heartbeat; failed simulations send Failure. ## 2. Add the time zone to the Flex query times (breaking) diff --git a/entrypoint.sh b/entrypoint.sh index be4d7ed..a2a756c 100644 --- a/entrypoint.sh +++ b/entrypoint.sh @@ -1,7 +1,7 @@ #!/bin/bash set -e -echo "ghostfolio-ibkr-sync version ${APP_VERSION:-dev}" +echo "ghostfolio-ibkr-sync build ${APP_VERSION:-dev}" if [ -n "$CRON" ]; then # Five numeric cron fields only; never interpolate an environment command. @@ -22,7 +22,7 @@ if [ -n "$CRON" ]; then exit 1 fi echo 'Running with validated cron schedule' - exec supercronic "$crontab_path" + exec supercronic -passthrough-logs -split-logs "$crontab_path" else echo "Running once (no CRON schedule set)" exec python -m ghostfolio_ibkr_sync diff --git a/ghostfolio_ibkr_sync/__main__.py b/ghostfolio_ibkr_sync/__main__.py index e70424f..19ea763 100644 --- a/ghostfolio_ibkr_sync/__main__.py +++ b/ghostfolio_ibkr_sync/__main__.py @@ -9,10 +9,15 @@ def run() -> int: + stdout = logging.StreamHandler(sys.stdout) + stdout.addFilter(lambda record: record.levelno < logging.WARNING) + stderr = logging.StreamHandler(sys.stderr) + stderr.setLevel(logging.WARNING) logging.basicConfig( level=getattr(logging, os.environ.get("LOG_LEVEL", "INFO").upper(), logging.INFO), format="%(asctime)s [%(levelname)s] %(message)s", datefmt="%Y-%m-%d %H:%M:%S", + handlers=[stdout, stderr], ) # urllib3 logs full request lines (incl. the IBKR token query param) at DEBUG. logging.getLogger("urllib3").setLevel(logging.WARNING) diff --git a/ghostfolio_ibkr_sync/activity_review.py b/ghostfolio_ibkr_sync/activity_review.py index fa6b22c..5130b00 100644 --- a/ghostfolio_ibkr_sync/activity_review.py +++ b/ghostfolio_ibkr_sync/activity_review.py @@ -897,11 +897,55 @@ def process_review(state: ReviewState, config: dict[str, Any], events: dict[str, apply_corporate_review_holds(config, refreshed, snapshot_rows(config)) publish_review(state, {"version": 1, "proposals": refreshed}) held = {item["source event"] for item in refreshed.values()} | unresolved_events - if held: - log.warning("Activity review holds %d source event(s); pending approvals remain in %s", len(held), state["path"]) + pending = {entry["source event"] for entry in state["journal"]["entries"].values() + if entry["status"] != "confirmed"} + report_review_outcome(state, config, refreshed, held, pending) return blocked, held + +def report_review_outcome(state: ReviewState, config: dict[str, Any], proposals: dict[str, Any], + held: set[str], uncertain: set[str]) -> None: + """Report decisions without changing approvals, holds or execution state.""" + from ghostfolio_ibkr_sync.outcome import ACTION_INSTRUCTIONS, record_action + + outcome = config.get("_corporate_outcome") + deferred: set[str] = set() + for key, item in sorted(proposals.items()): + event = item["source event"] + if event in uncertain: + reason = "review_uncertain" + elif item.get("execution path") == "historical maintenance": + reason = "review_maintenance" + elif (not item.get("actionable") or item.get("status") in + ("source unavailable", "target missing or unavailable") or QUALIFIED_EXECUTION is None): + reason = "review_evidence" + elif item.get("approve") == "yes": + deferred.add(event) + log.info("Activity review deferred automatic work: %s; container path=%s; section=%s; key=%s; " + "approval retained; hold remains; simulation=%s", event, state["path"], + item.get("section", "proposals"), key, bool(config.get("dry_run"))) + continue + else: + reason = "review_approval" + log.warning("User action required: %s; ACTIVITY_REVIEW_FILE container path=%s; section=%s; " + "key=%s; held event=%s; reason=%s; %s; only lowercase literal yes approves " + "an eligible normal-run proposal; simulation=%s", reason, state["path"], + item.get("section", "proposals"), key, event, item.get("status"), + ACTION_INSTRUCTIONS[reason], bool(config.get("dry_run"))) + record_action(outcome, reason, event) + for event in uncertain: + record_action(outcome, "review_uncertain", event) + log.warning("Review journal needs exact readback; container path=%s; held event=%s; %s", + state["journal_path"], event, ACTION_INSTRUCTIONS["review_uncertain"]) + if outcome is not None: + outcome["review_held"] = set(held) + outcome["deferred"] = deferred - uncertain + if held: + log.warning("Activity review holds %d unique source event(s); %d proposal(s); " + "ACTIVITY_REVIEW_FILE container path=%s", len(held), len(proposals), state["path"]) + + def apply_corporate_review_holds(config: dict[str, Any], proposals: dict[str, Any], rows: list[dict[str, Any]]) -> None: # Import here because corporate_actions reuses the private-file primitives. diff --git a/ghostfolio_ibkr_sync/app.py b/ghostfolio_ibkr_sync/app.py index 8c46ac5..9f6a7eb 100644 --- a/ghostfolio_ibkr_sync/app.py +++ b/ghostfolio_ibkr_sync/app.py @@ -25,9 +25,8 @@ source_events, ) from ghostfolio_ibkr_sync.config import ( - dry_run_requested, load_config, - load_notification_config, + load_notification_delivery, timezone_preview_reason, ) from ghostfolio_ibkr_sync.diagnostics import ( @@ -71,9 +70,13 @@ send_notification, ) from ghostfolio_ibkr_sync.outcome import ( + AccountResult, RunOutcome, + action_summary, is_token_rejection, + log_operational_result, new_run_outcome, + record_action, record_failure, ) from ghostfolio_ibkr_sync.sync import ( @@ -144,6 +147,11 @@ def process_account(config: dict[str, Any], ibkr_account_id: str, query_id: str, """ log.info("Processing IBKR account %s (Ghostfolio: %s)", ibkr_account_id, ghost_account_name) ok = True + outcome = config.get("_corporate_outcome") + account_result: AccountResult = {"simulated": bool(config.get("dry_run")), "accepted": None, + "duplicates": None, "held": None, "deferred": None, "cash": "unavailable"} + if outcome is not None: + outcome["account_results"][f"{ibkr_account_id}:{query_id}"] = account_result # Fetch the Flex Query report try: @@ -222,7 +230,8 @@ def process_account(config: dict[str, Any], ibkr_account_id: str, query_id: str, existing_dividend_comments, positions, query_id=query_id, diagnostic_preview=bool(config.get("dry_run")), review_holds=config.get("_review_holds"), - corporate_decision=config.get("_corporate_decisions", {}).get(ibkr_account_id)) + corporate_decision=config.get("_corporate_decisions", {}).get(ibkr_account_id), + outcome=outcome, account_result=account_result) except InvalidActivityIdentityError: return {}, False ok = ok and selection_ok @@ -232,6 +241,7 @@ def process_account(config: dict[str, Any], ibkr_account_id: str, query_id: str, # Create buys/dividends first. A server-skipped BUY cannot back a later SELL, # even in this account's batch. Dry-run uses proposed buys for simulation. + account_result["accepted"] = 0 phases = split_import_phases(activities) for phase in phases: if not phase: @@ -248,6 +258,7 @@ def process_account(config: dict[str, Any], ibkr_account_id: str, query_id: str, if not phase: continue imported, import_ok = ghost_import_activities(config, phase) + account_result["accepted"] = (account_result["accepted"] or 0) + len(imported) # Only accepted rows (or proposals in isolated simulation state) advance # the run. DIVIDEND identities must never act as trade IDs or holdings. record_accepted_activities(imported, positions, existing_trade_ids, @@ -259,12 +270,20 @@ def process_account(config: dict[str, Any], ibkr_account_id: str, query_id: str, try: cash_balance = parse_cash_report(xml_text) if cash_balance is not None: - if not ghost_update_cash_balance(config, ghost_account_id, cash_balance, - parse_base_currency(xml_text)): + config.setdefault("_cash_outcomes", {}).pop(ghost_account_id, None) + cash_ok = ghost_update_cash_balance(config, ghost_account_id, cash_balance, + parse_base_currency(xml_text)) + account_result["cash"] = config.get("_cash_outcomes", {}).get( + ghost_account_id, ("simulated" if config.get("dry_run") else "request reported successful") + if cash_ok else "uncertain/failed") + if not cash_ok: ok = False else: + account_result["cash"] = "not updated" + record_action(outcome, "cash_missing", f"{ibkr_account_id}:BASE_SUMMARY") log.warning("No BASE_SUMMARY cash balance found in report") except Exception as exc: + account_result["cash"] = config.get("_cash_outcomes", {}).get(ghost_account_id, "uncertain/failed") log.error("Failed to update cash balance: %s", exc) ok = False @@ -280,21 +299,27 @@ def select_account_activities( holdings_trace: dict[str, Any] | None = None, review_holds: set[str] | None = None, corporate_decision: ca.Decision | None = None, + outcome: RunOutcome | None = None, + account_result: AccountResult | None = None, ) -> tuple[list[Activity], dict[str, DividendContext], Unmapped, bool]: """Select without fetching or dispatching; mutate only manual-match state. Offline callers supply a captured clock and isolated holdings state. """ ok = True + if account_result is not None: + account_result["held"] = set() + account_result["deferred"] = set() review_holds = review_holds or set() date_is_current = date_is_current or activity_date_is_current # Parse trades and dividends trades: list[dict[str, Any]] = parse_trades(xml_text) - dividends = parse_cash_dividends(xml_text) + dividends = parse_cash_dividends(xml_text, outcome=outcome) if dividends is None: log.error("Flex Query %s has no Cash Transactions section: dividends not synced. " "Add Cash Transactions (Dividends, Payment In Lieu Of Dividends, " "Withholding Tax) to the query, see README", query_id) + record_action(outcome, "dividend_rejected", f"{ibkr_account_id}:cash_section") ok = False dividends = [] log.debug("Found %d trades and %d dividend payments in Flex Query report", @@ -322,9 +347,15 @@ def select_account_activities( unique, skipped_dup, conflicts = unique_trades(trades) if conflicts: + for trade_id in conflicts: + record_action(outcome, "trade_invalid", f"{ibkr_account_id}:{trade_id}") log.error("Conflicting repeated trade IDs excluded: %s", ", ".join(sorted(conflicts))) ok = False + for trade in trades: + isin = trade.get("isin", "") + if isin and isin not in mapping and resolve_symbol(isin, trade.get("symbol"), mapping): + record_action(outcome, "mapping_unverified", f"{ibkr_account_id}:{isin}") for trade in unique: trade_id = trade.get("tradeID", "") if format_versioned_activity_identity(ibkr_account_id, "TRADE", trade_id) in review_holds: @@ -349,11 +380,17 @@ def select_account_activities( trade_unmapped: Unmapped = {} activity = convert_trade_to_activity(trade, ghost_account_id, mapping, trade_unmapped) if activity is None or not date_is_current(activity) or not all(isfinite(activity[field]) for field in ("quantity", "unitPrice", "fee")): + if activity is None or not all(isfinite(activity[field]) for field in ("quantity", "unitPrice", "fee")): + record_action(outcome, "trade_invalid", f"{ibkr_account_id}:{trade_id}") + if activity is not None and not date_is_current(activity) and account_result is not None and account_result["deferred"] is not None: + account_result["deferred"].add(f"{ibkr_account_id}:{trade_id}") log.error("Trade %s cannot be converted safely; excluded before holdings checks", trade_id) skipped_other += 1 ok = False continue if corporate_decision is not None and ca.candidate_held(corporate_decision, activity): + if account_result is not None and account_result["held"] is not None: + account_result["held"].add(f"{ibkr_account_id}:{trade_id}") log.warning("Corporate-action BUY/SELL candidate held for %s", activity["symbol"]) ok = False continue @@ -364,7 +401,7 @@ def select_account_activities( "_activity": activity, "_unmapped": trade_unmapped}) trades = filter_trades_by_holdings(prepared_trades, positions, ghost_account_id, mapping, - existing_trade_ids, source_account=ibkr_account_id) + existing_trade_ids, source_account=ibkr_account_id, outcome=outcome) activities: list[Activity] = [] for kept in trades: activities.append(kept["_activity"]) @@ -379,12 +416,14 @@ def select_account_activities( findings = historical_dividend_comparison(dividends, ghost_account_id, mapping, positions, historical_dividend_report_window(xml_text), existing_dividend_comments) - report_counts = log_historical_dividend_comparison(findings, ghost_account_id) + report_counts = log_historical_dividend_comparison(findings, ghost_account_id, outcome=outcome) div_activities: list[Activity] = [] proposed_dividend_context: dict[str, DividendContext] = {} div_skipped_dup = 0 candidates = [] for div in dividends: + if div["amount"] > 0 and div.get("isin") and div["isin"] not in mapping: + record_action(outcome, "mapping_unverified", f"{ibkr_account_id}:{div['isin']}") if any(format_versioned_activity_identity(ibkr_account_id, "DIVIDEND", row.get("actionID", "")) in review_holds for row in div["payments"]): continue @@ -392,12 +431,13 @@ def select_account_activities( div, ghost_account_id, mapping, positions, existing_dividend_comments) if dedup == "not_payable": # Preserve existing diagnostics for reversals and tax-only rows. - convert_dividend_to_activity(div, ghost_account_id, mapping, unmapped) + convert_dividend_to_activity(div, ghost_account_id, mapping, unmapped, outcome=outcome) continue if dedup == "existing": div_skipped_dup += 1 continue if dedup == "partial": + record_action(outcome, "dividend_rejected", f"{ibkr_account_id}:{div['isin']}:{div['date']}") log.error("Dividend %s %s: partial versioned action coverage — not imported; " "account failed; automatic reruns will refuse while this group stays " "in the report window; explicit reviewed recovery required", @@ -405,13 +445,14 @@ def select_account_activities( ok = False continue if dedup == "action_conflict": + record_action(outcome, "dividend_rejected", f"{ibkr_account_id}:{div['isin']}:{div['date']}") log.error("Dividend %s %s: versioned action-context conflict or unknown overlap — " "not imported; account failed; manual review required", symbol or div.get("symbol") or "unknown", div["date"]) ok = False continue if dedup == "proximity": - log_dividend_proximity_skip(cast(str, symbol), div, cast(DividendEntry, matched)) + log_dividend_proximity_skip(cast(str, symbol), div, cast(DividendEntry, matched), outcome=outcome) div_skipped_dup += 1 continue candidates.append(div) @@ -433,11 +474,13 @@ def select_account_activities( if dividend_input_compatible(candidates): for div in candidates: proposed, partition_ok = convert_dividend_actions( - div, ghost_account_id, mapping, unmapped, dividends) + div, ghost_account_id, mapping, unmapped, dividends, outcome=outcome) if not partition_ok: ok = False continue if any(not date_is_current(activity) for activity in proposed): + if account_result is not None and account_result["deferred"] is not None: + account_result["deferred"].update(activity["comment"] for activity in proposed) log.error("Future or invalid dividend date; deferred until a later run") ok = False continue @@ -446,6 +489,10 @@ def select_account_activities( proposed_dividend_context[activity["comment"]] = dividend_context( {**activity, "isin": div["isin"]}, activity["date"]) + else: + for div in candidates: + record_action(outcome, "dividend_rejected", f"{ibkr_account_id}:{div['isin']}:{div['date']}") + if report_counts is None: log.info("New dividend activities: %d, duplicates skipped: %d", len(div_activities), div_skipped_dup) @@ -455,6 +502,8 @@ def select_account_activities( len(div_activities), div_skipped_dup, report_counts["to_check"], report_counts["informational"], report_counts["noise"]) + if account_result is not None: + account_result["duplicates"] = skipped_dup + div_skipped_dup activities.extend(div_activities) if holdings_trace is not None: holdings_trace.update({ @@ -491,59 +540,85 @@ def record_accepted_activities( existing_dividend_comments.add((activity["accountId"], comment)) + +def report_delivery(outcome: RunOutcome, status: str) -> None: + """Log one fixed transport result; never retry or change financial status.""" + outcome["delivery"] = status + warning = status not in ("sent", "not required") + log.log(logging.WARNING if warning else logging.INFO, "Notification result: %s%s", status, + "; inspect APPRISE_URLS/APPRISE_TIMEOUT and the local log" if warning else "") + + def finalize_run(outcome: RunOutcome, code: int) -> None: + """Finalize a completed run; interruptions propagate without a cycle result.""" + _finalize_notification(outcome, code) + log_operational_result(outcome, code) + + +def _finalize_notification(outcome: RunOutcome, code: int) -> None: """Send at most one notification for failures or a legacy-token reminder. - Ordinary warnings and successful simulations remain silent. Failures + Clean successful simulations remain silent. Failures notify in every mode, including invalid configuration. Delivery never changes the exit code. A reminder on a failed run joins its failure notification. """ + actions = action_summary(outcome) days = outcome.get("token_expiry_days") preview = outcome.get("forced_preview") corporate = outcome.get("corporate_notification") corporate_due = corporate is not None and bool(corporate["sent"]) maintenance_due = outcome.get("corporate_maintenance", False) - if code == 0 and not corporate_due and not maintenance_due and (dry_run_requested() or (days is None and not preview)): + if code == 0 and not actions and not corporate_due and not maintenance_due and days is None and not preview: + report_delivery(outcome, "not required") return label = "Failure" if code else ("Corporate actions" if corporate_due or maintenance_due else - ("Forced preview" if preview else "Token expiry")) + ("Forced preview" if preview else + "Token expiry" if days is not None else "User action")) try: - settings = load_notification_config() + settings, configuration = load_notification_delivery() if settings is None: + report_delivery(outcome, configuration) return if code: title, body = NOTIFY_TITLE, build_failure_summary(outcome) elif corporate_due or maintenance_due: title, body = "ghostfolio-ibkr-sync: corporate actions", "Corporate-action evidence summary." - if not dry_run_requested(): - if preview: - body += "\n" + build_forced_preview_summary(preview) - if days is not None: - body += "\n" + build_token_expiry_summary(days) + if preview: + body += "\n" + build_forced_preview_summary(preview) + if days is not None: + body += "\n" + build_token_expiry_summary(days) elif preview: title, body = FORCED_PREVIEW_NOTIFY_TITLE, build_forced_preview_summary(preview) if days is not None: body += "\n" + build_token_expiry_summary(days) + elif days is not None: + title, body = TOKEN_EXPIRY_NOTIFY_TITLE, build_token_expiry_summary(days) else: - title, body = TOKEN_EXPIRY_NOTIFY_TITLE, build_token_expiry_summary(cast(int, days)) + title, body = "ghostfolio-ibkr-sync: user action required", "User action required." if corporate_due and corporate is not None: body += "\n" + "\n".join(f"- corporate_actions: {key} ({value})" for key, value in corporate["counts"].items() if key in ("active_holds", "changed_findings", "resumptions") and type(value) is int) if maintenance_due: body += "\n- corporate_actions: maintenance_incomplete (1)" + if actions: + body += "\n" + actions status = send_notification(settings, title, body) except Exception: log.warning("%s notification could not be sent: internal error", label) + report_delivery(outcome, "failed") return + delivery = {"sent": "sent", "timeout": "timeout, delivery unconfirmed", + "invalid_destination": "invalid configuration"}.get(status, "failed") + report_delivery(outcome, delivery) if status == "sent": if corporate_due and corporate is not None: - ca.confirm_notification(cast(ca.NotificationPlan, corporate)) + ca.confirm_notification(corporate) log.info("%s notification sent", label) elif status == "timeout": log.warning("%s notification timed out (it may still arrive; not retried)", label) else: - log.warning("%s notification not delivered: %s", label, status) + log.warning("%s notification not delivered: %s", label, delivery) def run_sync(outcome: RunOutcome) -> int: @@ -554,6 +629,8 @@ def run_sync(outcome: RunOutcome) -> int: log.error("Configuration error: %s", exc) record_failure(outcome, "config", "invalid_configuration") return 1 + outcome["review_held"] = set() + outcome["deferred"] = set() state: ReviewState | None = None try: if config.get("activity_review_file"): @@ -645,6 +722,7 @@ def _run_configured_sync(config: dict[str, Any], outcome: RunOutcome) -> int: if reason: config["dry_run"] = True outcome["forced_preview"] = reason + record_action(outcome, "timezone_correction", "TZ") log.warning("%s: whole run forced to DRY_RUN, no Ghostfolio writes", reason) if config["dry_run"]: log.info("DRY RUN enabled (DRY_RUN) — no writes will be sent to Ghostfolio") @@ -659,6 +737,7 @@ def _run_configured_sync(config: dict[str, Any], outcome: RunOutcome) -> int: days = legacy_token_expiry_days(config) if days is not None: outcome["token_expiry_days"] = days + record_action(outcome, "token_renewal", "GHOST_TOKEN") if days: log.warning("Legacy GHOST_TOKEN expires in %d day(s); regenerate it or " "switch to GHOST_ACCESS_TOKEN", days) @@ -785,7 +864,6 @@ def reconcile_corporate(rows: list[dict[str, Any]] | None = None) -> None: len(failed_accounts), len(account_ids), ", ".join(failed_accounts)) return 1 - log.info("Sync complete") return 0 @@ -795,7 +873,8 @@ def main() -> int: SystemExit and KeyboardInterrupt propagate untouched: an interrupted run is not a completed run and sends no notification. """ - log.info("Starting IBKR to Ghostfolio sync (version %s)", os.environ.get("APP_VERSION", "dev")) + log.info("Starting IBKR to Ghostfolio sync") + log.info("ghostfolio-ibkr-sync build %s", os.environ.get("APP_VERSION", "dev")) if not os.environ.get("DRY_RUN", "").strip(): log.warning("DRY_RUN not set: simulation, nothing written") outcome = new_run_outcome() @@ -917,7 +996,14 @@ def refresh_corporate_decisions(config: dict[str, Any], mapping: dict[str, str], config["_corporate_decisions"] = decisions outcome = config.get("_corporate_outcome") if outcome is not None: + if any(decision["account_held"] or decision["symbols"] for decision in decisions.values()): + log.warning("Corporate-action holds require current source/destination proof; " + "CORPORATE_ACTION_STATE_FILE container path=%s; " + "CORPORATE_ACTION_EVIDENCE_FILE container path=%s; " + "restore the named evidence; do not reset state to release a hold", + config.get("corporate_action_state_file") or "unavailable", + config.get("corporate_action_evidence_file") or "not configured") if store is not None and store["maintenance_warning"]: outcome["corporate_maintenance"] = True - ca.prepare_notifications(outcome, config.get("corporate_action_state_file") + ca.prepare_notifications(cast(RunOutcome, outcome), config.get("corporate_action_state_file") if notification_state_safe else None, decisions) diff --git a/ghostfolio_ibkr_sync/config.py b/ghostfolio_ibkr_sync/config.py index f336636..6cc0002 100644 --- a/ghostfolio_ibkr_sync/config.py +++ b/ghostfolio_ibkr_sync/config.py @@ -175,7 +175,7 @@ def load_config() -> ConfigDict: } -def load_notifications() -> NotificationSettings | None: +def notification_configuration() -> tuple[NotificationSettings | None, str]: """Parse APPRISE_URLS and APPRISE_TIMEOUT; None means notifications are off. APPRISE_URLS is a JSON list of Apprise URL strings (blank or [] = off); @@ -185,7 +185,7 @@ def load_notifications() -> NotificationSettings | None: """ raw = os.environ.get("APPRISE_URLS", "") if not raw.strip(): - return None + return None, "disabled" try: urls = json.loads(raw) if len(raw) <= NOTIFY_PAYLOAD_MAX else None except ValueError: @@ -199,10 +199,23 @@ def load_notifications() -> NotificationSettings | None: for u in urls) or not 1 <= timeout <= APPRISE_TIMEOUT_MAX): log.warning("APPRISE_URLS or APPRISE_TIMEOUT is invalid: failure notifications disabled") - return None + return None, "invalid configuration" if not urls: - return None - return NotificationSettings(urls=tuple(urls), timeout=timeout) + return None, "disabled" + return NotificationSettings(urls=tuple(urls), timeout=timeout), "configured" + + + +def load_notifications() -> NotificationSettings | None: + """Return validated settings; invalid and disabled settings both disable send.""" + return notification_configuration()[0] + + +def load_notification_delivery() -> tuple[NotificationConfigDict | None, str]: + """Expose validation state without disclosing destination values.""" + settings, status = notification_configuration() + return ({"urls": list(settings.urls), "timeout": settings.timeout} + if settings is not None else None), status def load_notification_config() -> NotificationConfigDict | None: diff --git a/ghostfolio_ibkr_sync/corporate_actions.py b/ghostfolio_ibkr_sync/corporate_actions.py index 5b09fbf..d021ec4 100644 --- a/ghostfolio_ibkr_sync/corporate_actions.py +++ b/ghostfolio_ibkr_sync/corporate_actions.py @@ -31,6 +31,7 @@ IBKR_MAX_XML_BYTES, parse_ibkr_datetime, ) +from ghostfolio_ibkr_sync.outcome import RunOutcome, record_action log = logging.getLogger(__name__) @@ -1575,8 +1576,18 @@ def plan_notifications( def prepare_notifications( - outcome: dict[str, Any], filename: str | None, decisions: dict[str, Decision] + outcome: RunOutcome, filename: str | None, decisions: dict[str, Decision] ) -> None: + for source, decision in decisions.items(): + if decision["account_held"]: + record_action(outcome, "corporate_action", f"{source}:account_hold") + for event_key, event in decision["findings"].items(): + if event["held"] or event["conclusions"].get("history") == "bounded_cost_dates_differ": + record_action(outcome, "corporate_action", f"{source}:{event_key}") + log.warning("Corporate-action user action: source=%s event=%s; reasons=%s; " + "conclusions=%s; state container path=%s; inspect source/destination proof; " + "acknowledgment and state reset do not release holds", source, event_key, + event["reasons"], event["conclusions"], filename or "unavailable") plan = plan_notifications(filename, decisions) if plan["sent"]: outcome["corporate_notification"] = plan diff --git a/ghostfolio_ibkr_sync/diagnostics.py b/ghostfolio_ibkr_sync/diagnostics.py index 8cc7e8b..277aa34 100644 --- a/ghostfolio_ibkr_sync/diagnostics.py +++ b/ghostfolio_ibkr_sync/diagnostics.py @@ -23,6 +23,7 @@ _dividend_decimal, dividend_input_compatible, ) +from ghostfolio_ibkr_sync.outcome import RunOutcome, record_action from ghostfolio_ibkr_sync.sync import ( DIVIDEND_MATCH_DAYS, Positions, @@ -928,7 +929,26 @@ def _plain_counts(counts: Mapping[str, int]) -> str: return ",".join(f"{name}:{count}" for name, count in sorted(counts.items())) or "none" -def log_historical_dividend_comparison(findings: Sequence[Finding], account_id: str) -> dict[str, int]: + +# These are limits of the comparison, not a request for the user to invent +# evidence or change financially valid rows to fit a representation. +DIAGNOSTIC_LIMITATIONS = frozenset({ + "component partition identity not proven", "incomplete or conflicting component coverage", + "original JSON numeric evidence unavailable", "unsupported decimal range", "unsupported currency", + "destination outside proven source coverage", + "destination UTC date outside explicit report window; source evidence unavailable", + "inactive destination outside proven source coverage", +}) + + +def diagnostic_requires_action(finding: Finding) -> bool: + """Keep explicit comparison limits distinct from user-resolvable findings.""" + return (finding["readable"]["class"] == "to_check" + and finding["reason"] not in DIAGNOSTIC_LIMITATIONS) + + +def log_historical_dividend_comparison(findings: Sequence[Finding], account_id: str, + outcome: RunOutcome | None = None) -> dict[str, int]: """Report private diagnostic evidence without changing outcomes or persistence. Information blocks come first, then the summary line, then each item to check. @@ -940,6 +960,8 @@ def log_historical_dividend_comparison(findings: Sequence[Finding], account_id: unavailable: defaultdict[str, int] = defaultdict(int) to_check = [] for finding in findings: + if finding["reason"] in DIAGNOSTIC_LIMITATIONS: + log.info("Evidence limitation: %s; no user correction is requested", finding["reason"]) counts[finding["category"]] += 1 classes[finding["readable"]["class"]] += 1 for check in finding.get("identity_checks", []): @@ -957,7 +979,13 @@ def log_historical_dividend_comparison(findings: Sequence[Finding], account_id: log.info("[DRY RUN] Dividend report: %d to check, %d informational, %d noise", classes["to_check"], classes["information"], classes["noise"]) for finding in to_check: - log.info("%s", format_historical_dividend_finding(finding, account_id)) + log.warning("%s", format_historical_dividend_finding(finding, account_id)) + if diagnostic_requires_action(finding): + key = f"{account_id}:{finding['source']}:{','.join(sorted(finding['activity_ids']))}" + record_action(outcome, "dividend_check", key) + else: + log.warning("Evidence limitation: %s; comparison cannot certify this representation; " + "no user correction is requested", finding["reason"]) log.info("[DRY RUN] Historical dividends account=%s coverage=%d source_groups=%d " "unmatched_destination=%d categories=%s failed_checks=%s unavailable_checks=%s", account_id, len(findings), diff --git a/ghostfolio_ibkr_sync/ghostfolio.py b/ghostfolio_ibkr_sync/ghostfolio.py index a1d3f4c..7906b6a 100644 --- a/ghostfolio_ibkr_sync/ghostfolio.py +++ b/ghostfolio_ibkr_sync/ghostfolio.py @@ -616,6 +616,8 @@ def ghost_update_cash_balance( without a base currency, sends no PUT and returns False. A dry run still does the read-only GET and the check, so the preview matches a live run. """ + evidence = config.setdefault("_cash_outcomes", {}) + evidence[account_id] = "not updated" if not base_currency: log.error("Cash balance not updated for account %s: the report has no base currency " "(Account Information CurrencyPrimary missing from the Flex Query)", account_id) @@ -635,6 +637,7 @@ def ghost_update_cash_balance( return False log_lab_evidence("cash", {"accountId": account_id, "balance": balance, "currency": base_currency}) if config.get("dry_run"): + evidence[account_id] = "simulated" log.info("[DRY RUN] would set cash balance for account %s to %.2f (no PUT sent)", account_id, balance) return True @@ -646,6 +649,7 @@ def ghost_update_cash_balance( "name": account_data["name"], "platformId": account_data.get("platformId") or config.get("ghost_platform_id") or None, } + evidence[account_id] = "uncertain/failed" resp = requests.put(url, headers=ghost_headers(config["ghost_token"]), json=payload, timeout=30, allow_redirects=False) if ghost_response_is_redirect(resp): @@ -654,6 +658,7 @@ def ghost_update_cash_balance( if resp.status_code >= 400: log.error("Failed to update cash balance (%d): %s", resp.status_code, resp.text) return False + evidence[account_id] = "request reported successful" log.info("Updated cash balance for account %s to %.2f", account_id, balance) return True diff --git a/ghostfolio_ibkr_sync/ibkr.py b/ghostfolio_ibkr_sync/ibkr.py index b01348c..68c4d88 100644 --- a/ghostfolio_ibkr_sync/ibkr.py +++ b/ghostfolio_ibkr_sync/ibkr.py @@ -19,6 +19,8 @@ import requests +from ghostfolio_ibkr_sync.outcome import RunOutcome, record_action + log = logging.getLogger(__name__) IBKR_SEND_URL = ( @@ -340,7 +342,7 @@ def _dividend_cash_sum(rows: Iterable[Mapping[str, str]]) -> Decimal | None: return None -def parse_cash_dividends(xml_text: str) -> list[DividendGroup] | None: +def parse_cash_dividends(xml_text: str, outcome: RunOutcome | None = None) -> list[DividendGroup] | None: """Parse dividend payments from the Cash Transactions section. Structure: FlexQueryResponse > FlexStatements > FlexStatement > CashTransactions > CashTransaction @@ -370,6 +372,8 @@ def parse_cash_dividends(xml_text: str) -> list[DividendGroup] | None: amount = _parse_float(row.get("amount")) date_iso = parse_ibkr_datetime(row.get("dateTime", "")) if amount is None or not date_iso: + record_action(outcome, "dividend_rejected", + f"{row.get('accountId')}:{row.get('actionID')}:{row.get('dateTime')}") log.warning("Skipping cash transaction %s for %s: invalid amount or date", tx_type, row.get("symbol", "")) continue diff --git a/ghostfolio_ibkr_sync/outcome.py b/ghostfolio_ibkr_sync/outcome.py index c4ee9d6..7131bae 100644 --- a/ghostfolio_ibkr_sync/outcome.py +++ b/ghostfolio_ibkr_sync/outcome.py @@ -1,9 +1,13 @@ """Run-local failures and delivery bookkeeping; external summaries use fixed codes.""" -from typing import Any, NotRequired, TypedDict +import logging +from typing import TYPE_CHECKING, NotRequired, TypedDict import requests +if TYPE_CHECKING: + from ghostfolio_ibkr_sync.corporate_actions import NotificationPlan + class Failure(TypedDict): stage: str @@ -11,15 +15,29 @@ class Failure(TypedDict): ordinal: int | None +class AccountResult(TypedDict): + simulated: bool + accepted: int | None + duplicates: int | None + held: set[str] | None + deferred: set[str] | None + cash: str + + class RunOutcome(TypedDict): + account_results: dict[str, AccountResult] + actions: dict[str, set[str]] failures: list[Failure] dropped: int accounts_total: int accounts_failed: int token_expiry_days: NotRequired[int] forced_preview: NotRequired[str] - corporate_notification: NotRequired[dict[str, Any]] + corporate_notification: NotRequired[NotificationPlan] corporate_maintenance: NotRequired[bool] + delivery: NotRequired[str] + review_held: NotRequired[set[str]] + deferred: NotRequired[set[str]] # Allowed failure reasons per stage. External summaries use only these codes, @@ -47,7 +65,7 @@ def is_token_rejection(exc: BaseException) -> bool: def new_run_outcome() -> RunOutcome: """Return an empty run-local outcome.""" - return {"failures": [], "dropped": 0, "accounts_total": 0, "accounts_failed": 0} + return {"account_results": {}, "actions": {}, "failures": [], "dropped": 0, "accounts_total": 0, "accounts_failed": 0} def record_failure(outcome: RunOutcome, stage: str, reason: str, ordinal: object = None) -> None: @@ -56,7 +74,74 @@ def record_failure(outcome: RunOutcome, stage: str, reason: str, ordinal: object stage, reason = "unexpected", "unhandled_error" if isinstance(ordinal, bool) or not isinstance(ordinal, int) or ordinal < 1: ordinal = None + record_action(outcome, "processing_failure", f"{stage}:{ordinal}") if len(outcome["failures"]) >= MAX_OUTCOME_FAILURES: outcome["dropped"] += 1 return outcome["failures"].append({"stage": stage, "reason": reason, "ordinal": ordinal}) + + +# Private event keys never enter external summaries. Instructions and resolution +# proof are fixed text; callers supply only the observed code and private scope. +ACTION_INSTRUCTIONS = { + "processing_failure": "Inspect the fixed failure reason and private configuration or service log. Preserve financial recovery fences and verify uncertain requests before any resend. Resolution: current prerequisite and processing evidence pass.", + "cash_missing": "Inspect the Flex cash report configuration. Cash was not updated. Resolution: the current report supplies a valid BASE_SUMMARY and base currency.", + "corporate_action": "Inspect the corporate-action conclusions and restore the stated evidence or state. Holds require fresh source and destination agreement; acknowledgment or state reset cannot release them. Resolution: current conclusions prove safe continuation or the historical discrepancy is resolved.", + "token_renewal": "Regenerate GHOST_TOKEN or switch to GHOST_ACCESS_TOKEN. Resolution: a current credential is outside the configured expiry warning window.", + "timezone_correction": "Set TZ to the intended valid IANA timezone. Resolution: the next run validates TZ; simulation alone does not correct it.", + "dividend_rejected": "Inspect the rejected payable event and missing or conflicting Flex quantity, cash or tax evidence. Never invent payment or quantity. Resolution: complete consistent current source evidence supports the event.", + "dividend_reversal": "Compare the reversal with the original dividend in Ghostfolio and preserved Flex cash rows. Do not resend. Resolution: source and destination prove the reversal outcome or documented manual resolution.", + "dividend_signed_tax": "Inspect signed withholding, refunds and orphan tax against the original payment in Ghostfolio. Do not invent or resend a transaction. Resolution: current payment and tax evidence reconcile or documented manual resolution.", + "trade_invalid": "Inspect the omitted trade and required Flex fields. Do not invent a trade. Resolution: valid current source evidence permits safe conversion.", + "trade_manual": "Compare the source trade with the manual Ghostfolio activity before any reference repair. Resolution: verified reference ownership or documented manual resolution.", + "trade_position": "Compare current Ghostfolio holdings and source trades. Resolution: verified holdings support the intended sale; no automatic repair is authorized.", + "mapping_conflict": "Inspect the different-symbol holding in Ghostfolio and MAPPING_FILE. Resolution: the intended security and holding use a verified explicit mapping.", + "review_approval": "Read the exact proposal in ACTIVITY_REVIEW_FILE. Edit only approve between runs; lowercase literal yes approves an eligible normal-run correction. Resolution: current evidence confirms the intended correction or documented manual resolution.", + "review_evidence": "Inspect ACTIVITY_REVIEW_FILE and obtain the stated missing proof. yes cannot make blocked work executable. Resolution: current source and destination evidence qualify the proposal.", + "review_maintenance": "Follow the historical maintenance preparation guide and obtain separate operation authorization. Normal sync cannot execute it. Resolution: verified maintenance or documented manual resolution.", + "review_uncertain": "Inspect preserved journal and destination readback. Do not resend, delete or edit the journal. Resolution: exact readback confirms the request outcome.", + "mapping_unverified": "Verify the security in Ghostfolio or correct MAPPING_FILE. Resolution: an explicit mapping matches the intended destination security.", + "dividend_check": "Inspect the private dividend comparison and source evidence in Ghostfolio. Do not invent or resend a transaction. Resolution: current source and destination evidence agree or the stated discrepancy is resolved.", +} + + +def record_action(outcome: RunOutcome | None, reason: str, key: str) -> None: + """Register an explicit decision once per private event and reason.""" + if reason not in ACTION_INSTRUCTIONS: + raise ValueError("Unknown required-action reason") + if outcome is None: + return + keys = outcome["actions"].setdefault(reason, set()) + if key not in keys: + keys.add(key) + logging.getLogger(__name__).warning("User action required: %s; scope=%s; %s", + reason, key, ACTION_INSTRUCTIONS[reason]) + + +def action_summary(outcome: RunOutcome) -> str: + """Build external text from fixed codes and counts only.""" + return "\n".join(f"- {reason} ({len(keys)}): {ACTION_INSTRUCTIONS[reason]}" + for reason, keys in sorted(outcome["actions"].items()) + if reason in ACTION_INSTRUCTIONS and keys) + + +def log_operational_result(outcome: RunOutcome, code: int) -> None: + """Report only observed acceptance evidence; never imply balance readback.""" + log = logging.getLogger(__name__) + result = "failure" if code else "success with user action required" if outcome["actions"] else "success" + level = logging.ERROR if code else logging.WARNING if outcome["actions"] else logging.INFO + held = outcome.get("review_held") + deferred = outcome.get("deferred") + log.log(level, "Cycle result: %s; unique review-held events=%s; deferred automatic work=%s; notification=%s; " + "decision counts can overlap", result, len(held) if held is not None else "unavailable", + len(deferred) if deferred is not None else "unavailable", outcome.get("delivery", "unavailable")) + if not outcome["account_results"]: + log.info("Account evidence: imports=unavailable; duplicates=unavailable; cash=unavailable") + for key, account in outcome["account_results"].items(): + log.info("Account evidence %s: %s=%s; duplicates=%s; other held events=%s; " + "deferred automatic work=%s; cash=%s", key, + "simulated proposals" if account["simulated"] else "accepted imports", + account["accepted"] if account["accepted"] is not None else "unavailable", + account["duplicates"] if account["duplicates"] is not None else "unavailable", + len(account["held"]) if account["held"] is not None else "unavailable", + len(account["deferred"]) if account["deferred"] is not None else "unavailable", account["cash"]) diff --git a/ghostfolio_ibkr_sync/sync.py b/ghostfolio_ibkr_sync/sync.py index 5ced89e..002ab5f 100644 --- a/ghostfolio_ibkr_sync/sync.py +++ b/ghostfolio_ibkr_sync/sync.py @@ -31,6 +31,7 @@ parse_ibkr_datetime, ) from ghostfolio_ibkr_sync.models import DividendContext +from ghostfolio_ibkr_sync.outcome import RunOutcome, record_action log = logging.getLogger(__name__) @@ -103,7 +104,49 @@ def resolve_symbol(isin: str | None, ibkr_symbol: str | None, return None -def dividend_action_quantities(dividend: DividendGroup, quiet: bool = False + +DIVIDEND_REJECTION_ACTIONS = { + 'cash partition does not conserve source totals': "dividend_rejected", + 'conflicting payment association keys': "dividend_rejected", + 'conflicting payment day': "dividend_rejected", + 'contradictory cash currencies': "dividend_rejected", + 'contradictory component quantities': "dividend_rejected", + 'contradictory negative-net cash action': "dividend_rejected", + 'contradictory structured quantities': "dividend_rejected", + 'incompatible numeric cash amount': "dividend_rejected", + 'incompatible numeric cash totals': "dividend_rejected", + 'incompatible numeric price': "dividend_rejected", + 'incompatible numeric price or fee after scaling': "dividend_rejected", + 'incompatible numeric quantity': "dividend_rejected", + 'incompatible structured quantity': "dividend_rejected", + 'missing accrual association keys': "dividend_rejected", + 'missing net-positive quantity evidence': "dividend_rejected", + 'missing or conflicting withholding association': "dividend_rejected", + 'missing or contradictory payment reversal evidence': "dividend_rejected", + 'missing payment association keys': "dividend_rejected", + 'missing structured quantity evidence': "dividend_rejected", + 'no symbol resolved': "dividend_rejected", + 'source action split across cash groups': "dividend_rejected", + "signed zero-net withholding": "dividend_signed_tax", + "orphan withholding": "dividend_signed_tax", + "withholding refund": "dividend_signed_tax", + "orphan reversal": "dividend_reversal", +} + + +def dividend_scope(dividend: Mapping[str, Any]) -> str: + """Stable private scope shared by repeated observations of a source group.""" + accounts = sorted({row.get("accountId", "") for row in dividend.get("cash_rows", [])}) + return f"{','.join(accounts)}:{dividend.get('isin') or dividend.get('symbol')}:{dividend['date']}" + + +def report_dividend_rejection(dividend: DividendGroup, reason: str, outcome: RunOutcome | None) -> None: + """Report an enumerated decision, never infer an action from warning text.""" + record_action(outcome, DIVIDEND_REJECTION_ACTIONS[reason], dividend_scope(dividend)) + + +def dividend_action_quantities(dividend: DividendGroup, quiet: bool = False, + outcome: RunOutcome | None = None ) -> dict[tuple[str, str], Decimal] | None: """Return exact quantities keyed by source account/action for positive cash. @@ -114,6 +157,7 @@ def dividend_action_quantities(dividend: DividendGroup, quiet: bool = False """ def reject(reason: str) -> dict[tuple[str, str], Decimal] | None: if not quiet: + report_dividend_rejection(dividend, reason, outcome) log.warning("Dividend %s %s: %s — not imported", dividend["symbol"], dividend["date"], reason) return None @@ -172,18 +216,20 @@ def reject(reason: str) -> dict[tuple[str, str], Decimal] | None: return action_quantities -def dividend_structured_quantity(dividend: DividendGroup, quiet: bool = False) -> Decimal | None: +def dividend_structured_quantity(dividend: DividendGroup, quiet: bool = False, + outcome: RunOutcome | None = None) -> Decimal | None: """Return the common quantity of the historical aggregate projection. The action writer calls this only after partitioning each action. Keep the scalar helper for historical comparisons, never to combine new actions. """ - quantities = dividend_action_quantities(dividend, quiet=quiet) + quantities = dividend_action_quantities(dividend, quiet=quiet, outcome=outcome) if quantities is None: return None common_quantity = next(iter(quantities.values())) if any(quantity != common_quantity for quantity in quantities.values()): if not quiet: + report_dividend_rejection(dividend, "contradictory component quantities", outcome) log.warning("Dividend %s %s: contradictory component quantities — not imported", dividend["symbol"], dividend["date"]) return None @@ -330,7 +376,8 @@ def convert_trade_to_activity(trade: Mapping[str, str], ghost_account_id: str, def partition_dividend_actions(dividend: DividendGroup, - report_dividends: Sequence[DividendGroup] = () + report_dividends: Sequence[DividendGroup] = (), + outcome: RunOutcome | None = None ) -> list[DividendGroup] | None: """Partition cash exactly before quantity validation or fee clamping. @@ -338,6 +385,7 @@ def partition_dividend_actions(dividend: DividendGroup, separate boundary. Keep zero-net actions for cash accounting, not dispatch. """ def reject(reason: str) -> list[DividendGroup] | None: + report_dividend_rejection(dividend, reason, outcome) log.error("Dividend %s %s: %s — no action imported; account failed", dividend["symbol"], dividend["date"], reason) return None @@ -404,7 +452,8 @@ def reject(reason: str) -> list[DividendGroup] | None: def convert_dividend_actions(dividend: DividendGroup, ghost_account_id: str, mapping: Mapping[str, str], unmapped: Unmapped, - report_dividends: Sequence[DividendGroup] = () + report_dividends: Sequence[DividendGroup] = (), + outcome: RunOutcome | None = None ) -> tuple[list[Activity], bool]: """Prepare the complete historical group; never expose a partial proposal. @@ -412,7 +461,7 @@ def convert_dividend_actions(dividend: DividendGroup, ghost_account_id: str, rejections keep their local outcome. The scalar converter below remains the historical projection used to compare old financial values. """ - parts = partition_dividend_actions(dividend, report_dividends) + parts = partition_dividend_actions(dividend, report_dividends, outcome=outcome) if parts is None: return [], False activities = [] @@ -421,12 +470,13 @@ def convert_dividend_actions(dividend: DividendGroup, ghost_account_id: str, tax = _dividend_cash_sum([row for row in part["cash_rows"] if row.get("type") == WITHHOLDING_CASH_TYPE]) if tax: + report_dividend_rejection(part, "signed zero-net withholding", outcome) log.warning("Dividend %s %s: action %s nets to zero but carries signed " "withholding %s %s — not imported, check by hand", dividend["symbol"], dividend["date"], part["payments"][0]["actionID"], tax, dividend["currency"]) continue - activity = convert_dividend_to_activity(part, ghost_account_id, mapping, unmapped) + activity = convert_dividend_to_activity(part, ghost_account_id, mapping, unmapped, outcome=outcome) if activity is None: return [], True row = part["payments"][0] @@ -437,7 +487,8 @@ def convert_dividend_actions(dividend: DividendGroup, ghost_account_id: str, def convert_dividend_to_activity(dividend: DividendGroup, ghost_account_id: str, - mapping: Mapping[str, str], unmapped: Unmapped) -> Activity | None: + mapping: Mapping[str, str], unmapped: Unmapped, + outcome: RunOutcome | None = None) -> Activity | None: """Convert a parse_cash_dividends() entry to a Ghostfolio DIVIDEND activity. Returns None if the dividend cannot be processed. @@ -452,11 +503,13 @@ def convert_dividend_to_activity(dividend: DividendGroup, ghost_account_id: str, if amount <= 0: if amount < 0: + report_dividend_rejection(dividend, "orphan reversal", outcome) # a reversal dated apart from its payment: the original stays in Ghostfolio log.warning("Dividend %s %s: reversal of %.4g %s without a payment on the " "same day — not imported, check the original dividend by hand", ibkr_symbol, dividend["date"], -amount, currency) elif dividend["tax"]: + report_dividend_rejection(dividend, "orphan withholding", outcome) log.warning("Dividend %s %s: withholding tax %.4g %s without a dividend on the " "same day (tax correction?) — not imported, check by hand", ibkr_symbol, dividend["date"], dividend["tax"], currency) @@ -471,19 +524,22 @@ def convert_dividend_to_activity(dividend: DividendGroup, ghost_account_id: str, # setdefault: keep the description a trade for the same ISIN may have set unmapped.setdefault(isin, {"symbol": ibkr_symbol, "description": ""}) else: + report_dividend_rejection(dividend, "no symbol resolved", outcome) log.warning("No symbol resolved for dividend (ISIN: %s), skipping", isin) return None # IBKR reports withholding as a negative amount; a net refund is not a fee if dividend["tax"] > 0: + report_dividend_rejection(dividend, "withholding refund", outcome) log.warning("Dividend %s %s: net withholding refund %.4g %s — recorded as 0", ibkr_symbol, dividend["date"], dividend["tax"], currency) - quantity = dividend_structured_quantity(dividend) + quantity = dividend_structured_quantity(dividend, outcome=outcome) if quantity is None: return None qty = float(quantity) if not isfinite(qty) or qty <= 0: + report_dividend_rejection(dividend, "incompatible numeric quantity", outcome) log.warning("Dividend %s %s: incompatible numeric quantity — not imported", ibkr_symbol, dividend["date"]) return None @@ -493,6 +549,7 @@ def convert_dividend_to_activity(dividend: DividendGroup, ghost_account_id: str, tax = _dividend_cash_sum([row for row in cash_rows if row.get("type") == WITHHOLDING_CASH_TYPE]) if gross is None or tax is None or gross <= 0: + report_dividend_rejection(dividend, "incompatible numeric cash totals", outcome) log.warning("Dividend %s %s: incompatible numeric cash totals — not imported", ibkr_symbol, dividend["date"]) return None @@ -501,6 +558,7 @@ def convert_dividend_to_activity(dividend: DividendGroup, ghost_account_id: str, context.prec = 50 unit_price_decimal = gross / quantity except DecimalException: + report_dividend_rejection(dividend, "incompatible numeric price", outcome) log.warning("Dividend %s %s: incompatible numeric price — not imported", ibkr_symbol, dividend["date"]) return None @@ -520,6 +578,7 @@ def convert_dividend_to_activity(dividend: DividendGroup, ghost_account_id: str, if (not isfinite(unit_price) or unit_price <= 0 or not isfinite(wht) or wht < 0 or (fee_decimal > 0 and wht == 0)): + report_dividend_rejection(dividend, "incompatible numeric price or fee after scaling", outcome) log.warning("Dividend %s %s: incompatible numeric price or fee after scaling — not imported", ibkr_symbol, dividend["date"]) return None @@ -579,7 +638,8 @@ def _match_manual_entry(trade: Mapping[str, Any], entries: list[list[Any]] def filter_trades_by_holdings(trades: list[dict[str, Any]], positions: Positions, ghost_account_id: str, mapping: Mapping[str, str], - existing_trade_ids: set[str], *, source_account: str) -> list[dict[str, Any]]: + existing_trade_ids: set[str], *, source_account: str, + outcome: RunOutcome | None = None) -> list[dict[str, Any]]: """Drop trades whose import would duplicate a manual entry or drive a Ghostfolio position negative. @@ -640,6 +700,7 @@ def match_one(trade: Mapping[str, Any], ticker: str, isin: str, log.debug("Not importing sell %s of %s: already entered manually in Ghostfolio", trade_id, ticker) else: + record_action(outcome, "trade_manual", f"{source_account}:{trade_id}") log.warning("Not importing buy %s of %s (qty %.4g, %s): matches a manual Ghostfolio " "buy of %.4g on %s — set that entry's comment to %s to silence " "this", trade_id, ticker, abs(qty), trade.get("dateTime", ""), @@ -663,10 +724,15 @@ def match_one(trade: Mapping[str, Any], ticker: str, isin: str, ids = [t.get("tradeID", "") for t in fills] if len(fills) > 1 and match_one(summed, ticker, isin, side)[0] == "match": (manual_recorded if side == "SELL" else manual_buys).update(ids) + if side == "BUY": + for trade_id in ids: + record_action(outcome, "trade_manual", f"{source_account}:{trade_id}") log.log(logging.DEBUG if side == "SELL" else logging.WARNING, "Not importing %ss %s of %s: fills sum to a manual Ghostfolio %s", side.lower(), ",".join(ids), ticker, side.lower()) continue + for trade_id in ids: + record_action(outcome, "trade_manual", f"{source_account}:{trade_id}") ambiguous.update(ids) log.warning("Not importing %s(s) %s of %s (qty %.4g, %s): a manual Ghostfolio %s " "within 2 days has another quantity — check for a duplicate by hand", @@ -684,6 +750,7 @@ def match_one(trade: Mapping[str, Any], ticker: str, isin: str, others = sorted(sym for sym in positions["isin_symbols"].get((ghost_account_id, info["isin"]), ()) if abs(qty_by_symbol.get((ghost_account_id, sym), 0.0)) >= 0.001) if others: + record_action(outcome, "mapping_conflict", f"{source_account}:{info['isin']}") rejected.add(ticker) log.warning("Not importing sells for %s: Ghostfolio holds this ISIN under %s — " "add mapping symbol_mapping: '%s: %s'", @@ -700,6 +767,7 @@ def match_one(trade: Mapping[str, Any], ticker: str, isin: str, log.debug("Not importing %s: net new trades %.4g but Ghostfolio holds no position " "(sold manually or never tracked)", label, info["pending"]) else: + record_action(outcome, "trade_position", f"{source_account}:{ticker}") log.error("Not importing %s: Ghostfolio holds %.4g, net new trades %.4g would go " "negative — fix the position in Ghostfolio manually", label, held, info["pending"]) @@ -758,7 +826,8 @@ def _source_dividend_fee(div: DividendGroup) -> Decimal | None: return None if tax is None else max(Decimal(0), tax.copy_negate()) -def log_dividend_proximity_skip(symbol: str, div: DividendGroup, entry: DividendEntry) -> None: +def log_dividend_proximity_skip(symbol: str, div: DividendGroup, entry: DividendEntry, + outcome: RunOutcome | None = None) -> None: """Make a proximity skip visible: INFO when gross agrees, else one WARNING.""" source_text, text, agree = _proximity_amounts(div, entry) fees = f"source fee {_source_dividend_fee(div)}, existing fee {entry[3]}" @@ -767,6 +836,7 @@ def log_dividend_proximity_skip(symbol: str, div: DividendGroup, entry: Dividend "the same payment (not imported); amounts agree (%s; %s)", symbol, div["date"], entry[0], text, fees) return + record_action(outcome, "dividend_check", dividend_scope(div)) log.warning("Dividend %s %s: Ghostfolio has a dividend of this symbol on %s, skipped as " "the same payment (not imported), but gross differs: source %s, existing %s; %s. " "Check IBKR; if the payment is distinct, add it by hand in Ghostfolio", diff --git a/openspec/changes/clarify-operator-actions/.openspec.yaml b/openspec/changes/archive/2026-10-27-clarify-operator-actions/.openspec.yaml similarity index 100% rename from openspec/changes/clarify-operator-actions/.openspec.yaml rename to openspec/changes/archive/2026-10-27-clarify-operator-actions/.openspec.yaml diff --git a/openspec/changes/clarify-operator-actions/design.md b/openspec/changes/archive/2026-10-27-clarify-operator-actions/design.md similarity index 100% rename from openspec/changes/clarify-operator-actions/design.md rename to openspec/changes/archive/2026-10-27-clarify-operator-actions/design.md diff --git a/openspec/changes/clarify-operator-actions/proposal.md b/openspec/changes/archive/2026-10-27-clarify-operator-actions/proposal.md similarity index 100% rename from openspec/changes/clarify-operator-actions/proposal.md rename to openspec/changes/archive/2026-10-27-clarify-operator-actions/proposal.md diff --git a/openspec/changes/clarify-operator-actions/specs/activity-correction-review/spec.md b/openspec/changes/archive/2026-10-27-clarify-operator-actions/specs/activity-correction-review/spec.md similarity index 100% rename from openspec/changes/clarify-operator-actions/specs/activity-correction-review/spec.md rename to openspec/changes/archive/2026-10-27-clarify-operator-actions/specs/activity-correction-review/spec.md diff --git a/openspec/changes/clarify-operator-actions/specs/connector-runtime/spec.md b/openspec/changes/archive/2026-10-27-clarify-operator-actions/specs/connector-runtime/spec.md similarity index 100% rename from openspec/changes/clarify-operator-actions/specs/connector-runtime/spec.md rename to openspec/changes/archive/2026-10-27-clarify-operator-actions/specs/connector-runtime/spec.md diff --git a/openspec/changes/clarify-operator-actions/specs/corporate-action-detection/spec.md b/openspec/changes/archive/2026-10-27-clarify-operator-actions/specs/corporate-action-detection/spec.md similarity index 100% rename from openspec/changes/clarify-operator-actions/specs/corporate-action-detection/spec.md rename to openspec/changes/archive/2026-10-27-clarify-operator-actions/specs/corporate-action-detection/spec.md diff --git a/openspec/changes/clarify-operator-actions/tasks.md b/openspec/changes/archive/2026-10-27-clarify-operator-actions/tasks.md similarity index 93% rename from openspec/changes/clarify-operator-actions/tasks.md rename to openspec/changes/archive/2026-10-27-clarify-operator-actions/tasks.md index ba06dc0..5d49bd6 100644 --- a/openspec/changes/clarify-operator-actions/tasks.md +++ b/openspec/changes/archive/2026-10-27-clarify-operator-actions/tasks.md @@ -4,28 +4,28 @@ All paths below are relative to this repository. Specification paths are relativ ## 1. Review actions and notification delivery -- [ ] 1.1 Report a pending normal-run review as required user action and send one reminder per completed live or simulated run. Reuse typed run-local outcome records and existing finalization. Cover unchanged runs, duplicate observations, resolution and clean silence. Update the repeat policy in docs/configuration.md. Files: ghostfolio_ibkr_sync/outcome.py, app.py, activity_review.py, notify.py; tests/test_finalize.py. Spec: specs/connector-runtime/spec.md — Notifications; scenarios Repeated required action and resolution, One private summary across all reasons. Check: `python -m pytest -q tests/test_finalize.py -k 'operator_action_reminders or operator_action_summary'`. -- [ ] 1.2 Make every completed run's notification result explicit. Cover disabled/invalid settings, reported send success, worker failure, timeout, interruption, redaction and no recursive send. Keep the worker deadline, isolation and exits. Update delivery-state examples in docs/configuration.md. Files: ghostfolio_ibkr_sync/config.py, app.py, notify.py, outcome.py; tests/test_finalize.py, tests/test_notify.py. Spec: specs/connector-runtime/spec.md — Notifications; scenario Explicit delivery result. Check: `python -m pytest -q tests/test_finalize.py tests/test_notify.py -k operator_action_delivery`. -- [ ] 1.3 Give safe instructions for normal approval, invalid/stale choice, missing proof, historical maintenance and uncertain requests. Distinguish unique held events from proposals and already approved automatic deferral. Show the configured container path and document volume resolution. Files: ghostfolio_ibkr_sync/activity_review.py, app.py; tests/test_activity_review.py, tests/test_finalize.py; docs/activity-review.md. Specs: specs/activity-correction-review/spec.md — Verified correction and dependent import state, Safe review instructions; both operator-action scenarios. Check: `python -m pytest -q tests/test_activity_review.py tests/test_finalize.py -k operator_action_review`. +- [x] 1.1 Report a pending normal-run review as required user action and send one reminder per completed live or simulated run. Reuse typed run-local outcome records and existing finalization. Cover unchanged runs, duplicate observations, resolution and clean silence. Update the repeat policy in docs/configuration.md. Files: ghostfolio_ibkr_sync/outcome.py, app.py, activity_review.py, notify.py; tests/test_finalize.py. Spec: specs/connector-runtime/spec.md — Notifications; scenarios Repeated required action and resolution, One private summary across all reasons. Check: `python -m pytest -q tests/test_finalize.py -k 'operator_action_reminders or operator_action_summary'`. +- [x] 1.2 Make every completed run's notification result explicit. Cover disabled/invalid settings, reported send success, worker failure, timeout, interruption, redaction and no recursive send. Keep the worker deadline, isolation and exits. Update delivery-state examples in docs/configuration.md. Files: ghostfolio_ibkr_sync/config.py, app.py, notify.py, outcome.py; tests/test_finalize.py, tests/test_notify.py. Spec: specs/connector-runtime/spec.md — Notifications; scenario Explicit delivery result. Check: `python -m pytest -q tests/test_finalize.py tests/test_notify.py -k operator_action_delivery`. +- [x] 1.3 Give safe instructions for normal approval, invalid/stale choice, missing proof, historical maintenance and uncertain requests. Distinguish unique held events from proposals and already approved automatic deferral. Show the configured container path and document volume resolution. Files: ghostfolio_ibkr_sync/activity_review.py, app.py; tests/test_activity_review.py, tests/test_finalize.py; docs/activity-review.md. Specs: specs/activity-correction-review/spec.md — Verified correction and dependent import state, Safe review instructions; both operator-action scenarios. Check: `python -m pytest -q tests/test_activity_review.py tests/test_finalize.py -k operator_action_review`. ## 2. Remaining required-action sources Before implementation, map each existing reason string to one of: repeated user action with proof of resolution, automatic retry/deferred work, or informational evidence limitation. The mapping is part of the tests; no warning is promoted by severity alone. -- [ ] 2.1 Include unverified symbol fallback, different-symbol holdings, manual trade conflicts and invalid omitted trade evidence in the required-action inventory. Re-evaluate the relevant current evidence even after ordinary deduplication, without changing selection. Cover two accounts, repeated events, recovered retries and normal exclusions. Document the reason/action examples in docs/configuration.md. Files: ghostfolio_ibkr_sync/sync.py, ibkr.py, app.py, outcome.py; tests/test_filters.py, tests/test_conversion.py, tests/test_orchestration.py, tests/test_finalize.py. Spec: specs/connector-runtime/spec.md — Required user actions, Mapping and duplicate reconciliation; scenarios Decision inventory and unchanged financial behavior, Required-action reporting compatibility. Check: `python -m pytest -q tests -k 'operator_action_inventory and (trade or mapping or retry)'`. -- [ ] 2.2 Report payable dividend rejection, signed-tax and reversal cases, missing/conflicting quantity or tax evidence, orphan payment/tax cases needing investigation and discrepant proximity skips as required actions. Keep ordinary zero-net/nonpayable exclusions, automatically retried evidence maintenance and evidence-limited diagnostics distinct. Enumerate each current rejection reason and its action/proof mapping; do not infer a reminder from a generic warning. Preserve all local rejection outcomes and payloads. Update dividend action examples in docs/configuration.md. Files: ghostfolio_ibkr_sync/ibkr.py, sync.py, app.py; tests/test_conversion.py, tests/test_filters.py, tests/test_orchestration.py, tests/test_finalize.py. Spec: specs/connector-runtime/spec.md — Required user actions, Enriched dividend input and local rejection, Mapping and duplicate reconciliation. Check: `python -m pytest -q tests -k 'operator_action_inventory and dividend'`. -- [ ] 2.3 Emit diagnostic items to check at WARNING and include them once in the consolidated simulation notification. Keep informational/noise classes, quantized values, match proposals and single safe action lines. Update simulation behavior and examples in docs/configuration.md. Files: ghostfolio_ibkr_sync/diagnostics.py, app.py; tests/test_dividend_diagnostics.py, tests/test_check_dividend_report.py, tests/test_finalize.py. Spec: specs/connector-runtime/spec.md — Diagnostic execution and visibility, Readable dividend report, Proposed dividend match. Check: `python -m pytest -q tests/test_dividend_diagnostics.py tests/test_check_dividend_report.py tests/test_finalize.py -k 'operator_action_diagnostic or operator_action_inventory'`. -- [ ] 2.4 Consolidate corporate-action user actions with other reasons while preserving first/change/resumption notifications for findings that need no action. Add required token-renewal and timezone-correction notifications in simulation. Cover fingerprint loss, repeated active actions and mixed failures without extra dispatch. Update docs/configuration.md and the relevant README notification text. Files: ghostfolio_ibkr_sync/corporate_actions.py, app.py, notify.py; tests/test_corporate_actions.py, tests/test_token_expiry.py, tests/test_finalize.py. Specs: specs/corporate-action-detection/spec.md — Consolidated corporate-action reporting; specs/connector-runtime/spec.md — Simulation, Notifications, Required user actions. Check: `python -m pytest -q tests/test_corporate_actions.py tests/test_token_expiry.py tests/test_finalize.py -k 'operator_action_corporate or operator_action_prerequisite'`. +- [x] 2.1 Include unverified symbol fallback, different-symbol holdings, manual trade conflicts and invalid omitted trade evidence in the required-action inventory. Re-evaluate the relevant current evidence even after ordinary deduplication, without changing selection. Cover two accounts, repeated events, recovered retries and normal exclusions. Document the reason/action examples in docs/configuration.md. Files: ghostfolio_ibkr_sync/sync.py, ibkr.py, app.py, outcome.py; tests/test_filters.py, tests/test_conversion.py, tests/test_orchestration.py, tests/test_finalize.py. Spec: specs/connector-runtime/spec.md — Required user actions, Mapping and duplicate reconciliation; scenarios Decision inventory and unchanged financial behavior, Required-action reporting compatibility. Check: `python -m pytest -q tests -k 'operator_action_inventory and (trade or mapping or retry)'`. +- [x] 2.2 Report payable dividend rejection, signed-tax and reversal cases, missing/conflicting quantity or tax evidence, orphan payment/tax cases needing investigation and discrepant proximity skips as required actions. Keep ordinary zero-net/nonpayable exclusions, automatically retried evidence maintenance and evidence-limited diagnostics distinct. Enumerate each current rejection reason and its action/proof mapping; do not infer a reminder from a generic warning. Preserve all local rejection outcomes and payloads. Update dividend action examples in docs/configuration.md. Files: ghostfolio_ibkr_sync/ibkr.py, sync.py, app.py; tests/test_conversion.py, tests/test_filters.py, tests/test_orchestration.py, tests/test_finalize.py. Spec: specs/connector-runtime/spec.md — Required user actions, Enriched dividend input and local rejection, Mapping and duplicate reconciliation. Check: `python -m pytest -q tests -k 'operator_action_inventory and dividend'`. +- [x] 2.3 Emit diagnostic items to check at WARNING and include them once in the consolidated simulation notification. Keep informational/noise classes, quantized values, match proposals and single safe action lines. Update simulation behavior and examples in docs/configuration.md. Files: ghostfolio_ibkr_sync/diagnostics.py, app.py; tests/test_dividend_diagnostics.py, tests/test_check_dividend_report.py, tests/test_finalize.py. Spec: specs/connector-runtime/spec.md — Diagnostic execution and visibility, Readable dividend report, Proposed dividend match. Check: `python -m pytest -q tests/test_dividend_diagnostics.py tests/test_check_dividend_report.py tests/test_finalize.py -k 'operator_action_diagnostic or operator_action_inventory'`. +- [x] 2.4 Consolidate corporate-action user actions with other reasons while preserving first/change/resumption notifications for findings that need no action. Add required token-renewal and timezone-correction notifications in simulation. Cover fingerprint loss, repeated active actions and mixed failures without extra dispatch. Update docs/configuration.md and the relevant README notification text. Files: ghostfolio_ibkr_sync/corporate_actions.py, app.py, notify.py; tests/test_corporate_actions.py, tests/test_token_expiry.py, tests/test_finalize.py. Specs: specs/corporate-action-detection/spec.md — Consolidated corporate-action reporting; specs/connector-runtime/spec.md — Simulation, Notifications, Required user actions. Check: `python -m pytest -q tests/test_corporate_actions.py tests/test_token_expiry.py tests/test_finalize.py -k 'operator_action_corporate or operator_action_prerequisite'`. ## 3. Operational result and readable logs Task 2.2 SHALL enumerate signed-tax, reversal, orphan and zero-net rejection reasons. Ordinary nonpayable exclusions, automatic retries and evidence-limited diagnostics SHALL remain distinct. A reminder SHALL name its observable resolution proof. -- [ ] 3.1 Replace ambiguous completion text with the three defined cycle results. Carry accepted or simulated imports, duplicates, unique holds, deferred work and per-account cash evidence to finalization. Include missing BASE_SUMMARY and early failures without inventing zeros or readback. Update result examples in docs/configuration.md and README.md. Files: ghostfolio_ibkr_sync/app.py, outcome.py, notify.py; ghostfolio.py only if existing acceptance evidence is not exposed; tests/test_orchestration.py, tests/test_finalize.py. Spec: specs/connector-runtime/spec.md — Final operational result, Required user actions. Check: `python -m pytest -q tests/test_orchestration.py tests/test_finalize.py -k operator_action_result`. -- [ ] 3.2 Route DEBUG/INFO to stdout and WARNING or above to stderr without duplication. Enable pinned Supercronic passthrough and stream splitting. Log build provenance with `ghostfolio-ibkr-sync build `; do not call the value a release version when the stable tag reuses a qualified RC digest. Preserve cron validation, lifecycle behavior, credential suppression and LAB_EVIDENCE JSON. Document the application level versus transport stream and clean scheduled examples in docs/configuration.md. Files: ghostfolio_ibkr_sync/__main__.py, entrypoint.sh; tests/test_logging.py, tests/test_entrypoint.py, tests/test_run_evidence.py. Spec: specs/connector-runtime — Clear application log framing, Minimal native lab evidence. Check: `python -m pytest -q tests/test_entrypoint.py tests/test_run_evidence.py tests/test_logging.py -k operator_action_logging`; capture built-image stdout/stderr separately and prove no duplicate or nested application framing. +- [x] 3.1 Replace ambiguous completion text with the three defined cycle results. Carry accepted or simulated imports, duplicates, unique holds, deferred work and per-account cash evidence to finalization. Include missing BASE_SUMMARY and early failures without inventing zeros or readback. Update result examples in docs/configuration.md and README.md. Files: ghostfolio_ibkr_sync/app.py, outcome.py, notify.py; ghostfolio.py only if existing acceptance evidence is not exposed; tests/test_orchestration.py, tests/test_finalize.py. Spec: specs/connector-runtime/spec.md — Final operational result, Required user actions. Check: `python -m pytest -q tests/test_orchestration.py tests/test_finalize.py -k operator_action_result`. +- [x] 3.2 Route DEBUG/INFO to stdout and WARNING or above to stderr without duplication. Enable pinned Supercronic passthrough and stream splitting. Log build provenance with `ghostfolio-ibkr-sync build `; do not call the value a release version when the stable tag reuses a qualified RC digest. Preserve cron validation, lifecycle behavior, credential suppression and LAB_EVIDENCE JSON. Document the application level versus transport stream and clean scheduled examples in docs/configuration.md. Files: ghostfolio_ibkr_sync/__main__.py, entrypoint.sh; tests/test_logging.py, tests/test_entrypoint.py, tests/test_run_evidence.py. Spec: specs/connector-runtime — Clear application log framing, Minimal native lab evidence. Check: `python -m pytest -q tests/test_entrypoint.py tests/test_run_evidence.py tests/test_logging.py -k operator_action_logging`; capture built-image stdout/stderr separately and prove no duplicate or nested application framing. ## 4. Integrated acceptance -- [ ] 4.1 Exercise combined review, mapping, diagnostics, corporate actions, cash, delivery failure and clean resolution across consecutive mocked cycles. Require nonempty collection for every new acceptance selector, one redacted dispatch at most and unchanged financial request sets/exits. Remove obsolete notification-silence assertions and documentation contradictions. Review shared-helper changes for the separate DEGIRO propagation flag. Files: relevant tests from groups 1–3 and their user documentation. Specs: all three deltas and their combined scenarios. Checks: `python -m pytest -q tests -k operator_action`; `bash scripts/check.sh`; `docker build -t ibkr-operator-actions:check .` then `bash scripts/check_container_runtime.sh ibkr-operator-actions:check`; `rg -n -A 12 'User action required|Notification result|container path' docs/configuration.md docs/activity-review.md README.md`. Compare collected acceptance tests against each scenario. Use only the isolated existing runtime smoke check; no lab or production replay in this task. +- [x] 4.1 Exercise combined review, mapping, diagnostics, corporate actions, cash, delivery failure and clean resolution across consecutive mocked cycles. Require nonempty collection for every new acceptance selector, one redacted dispatch at most and unchanged financial request sets/exits. Remove obsolete notification-silence assertions and documentation contradictions. Review shared-helper changes for the separate DEGIRO propagation flag. Files: relevant tests from groups 1–3 and their user documentation. Specs: all three deltas and their combined scenarios. Checks: `python -m pytest -q tests -k operator_action`; `bash scripts/check.sh`; `docker build -t ibkr-operator-actions:check .` then `bash scripts/check_container_runtime.sh ibkr-operator-actions:check`; `rg -n -A 12 'User action required|Notification result|container path' docs/configuration.md docs/activity-review.md README.md`. Compare collected acceptance tests against each scenario. Use only the isolated existing runtime smoke check; no lab or production replay in this task. One implementation item will own this checklist after artifact approval. Each task is checked in its implementation commit. Keep the branch for the same change. The final implementation commit archives the change after acceptance. Publication, Pullfrog review and release retain their existing gates. A release child owns exact-candidate lab and stable-publication proof; this planning work authorizes no publication or deployment. diff --git a/openspec/specs/activity-correction-review/spec.md b/openspec/specs/activity-correction-review/spec.md index dd216cd..95f44bd 100644 --- a/openspec/specs/activity-correction-review/spec.md +++ b/openspec/specs/activity-correction-review/spec.md @@ -97,7 +97,7 @@ Before each dispatch, fresh unredacted destination evidence SHALL match the appr Before the request, durably record its intent and full before/after evidence in the journal. After the request, readback SHALL prove every approved value and every preserved destination field. A successful status alone SHALL NOT confirm completion. Changes to numeric values or UTC instants, changed owner/account, tags, unsupported profile effects or financial differences SHALL fail verification. Equivalent numeric and timestamp representations defined above SHALL NOT fail verification. Comments and preserved note text SHALL remain exact; trimming is not an equivalent representation. Only confirmed corrections SHALL affect real holdings/identity state. Rebuild that state before ordinary selection and sales checks; replacement of a BUY SHALL NOT add its full quantity a second time. -A known or uncertain correction failure SHALL stop later financial writes for the same account, including imports, cash and platform updates. Earlier confirmed corrections SHALL remain explicit. Later accounts SHALL continue if their own evidence and files remain valid. Ordinary import failures outside this correction path SHALL retain existing independent-cash behavior. Pending proposals alone SHALL warn without failing the account; correction failures SHALL fail it and enter the existing single Failure summary without financial details. +A known or uncertain correction failure SHALL stop later financial writes for the same account, including imports, cash and platform updates. Earlier confirmed corrections SHALL remain explicit. Later accounts SHALL continue if their own evidence and files remain valid. Ordinary import failures outside this correction path SHALL retain existing independent-cash behavior. Pending proposals alone SHALL warn without failing the account. Those requiring intervention SHALL join the repeated consolidated notification under connector-runtime. Approved proposals deferred solely by the qualified execution limit SHALL not request another approval. Correction failures SHALL fail the account and enter the existing single Failure summary without financial details. Correction PUTs SHALL be sequential. Each request SHALL complete exact readback before dispatching the next. Activation qualification SHALL establish a finite per-run correction limit and any minimum inter-request delay for the exact destination image, including market-data/background effects. Record the qualified limits in the shipped configuration and operator guidance; do not invent a safe production default from mocks. Stop at the qualified limit without failing the account, preserve unchanged approvals and proposals for the next run, and retain their related import holds. No interactive prompt or separate bulk command SHALL replace normal-run processing. @@ -113,6 +113,10 @@ Correction PUTs SHALL be sequential. Each request SHALL complete exact readback - **WHEN** `.venv/bin/python -m pytest -q tests/test_activity_review.py -k review_holdings` corrects BUY/SELL quantities or symbols before selecting later trades - **THEN** positions and reference indexes reflect replacement effects once, no corrected event is imported again, and subsequent sales use verified refreshed holdings. +#### Scenario: Review actions do not change correction authority +- **WHEN** `python -m pytest -q tests/test_activity_review.py -k operator_action_review` covers pending, blocked, historical, uncertain and budget-deferred proposals +- **THEN** required intervention notifies without authorizing corrections; automatic budget deferral does not request another approval. + ### Requirement: Bounded read-only recovery after uncertainty A timeout, interrupted connection, crash with pending intent or unavailable readback SHALL retain an unresolved journal entry. Recovery SHALL read the activity immediately, then after waits of 2, 5 and 10 seconds while its state remains unchanged or unreadable. Reads SHALL have finite request timeouts and no automatic retry loop. The waits are a connector budget, not proof that a delayed write has settled. @@ -150,3 +154,17 @@ README SHALL document the ENV opt-in, persistent-volume permissions, YAML exampl - **THEN** offline results and actual image evidence are distinguished; missing preservation, recovery or reader-rollback proof blocks real enablement rather than weakening the approved contract. All test commands above name acceptance tests to implement. They are not evidence of existing feature delivery. Baseline evidence is the README, existing filter/index tests and the pinned update controller/service/DTO cited in connector-runtime. + +### Requirement: Safe review instructions + +Each review report requiring user intervention SHALL identify ACTIVITY_REVIEW_FILE and its actual configured container path. Name the generated section and proposal key where available, reason, held scope and safe action. Count unique held source events separately from candidates/proposals. Group repeated reasons without losing affected keys or distinct execution paths. Do not invent a Docker host path or inspect the Docker socket. User documentation SHALL show volume-mapping resolution. + +For eligible unapproved normal-run proposals, instruct the user to read current/proposed values and execution path, then edit only approve between runs if the correction is intended. State that only lowercase literal yes approves that exact eligible proposal. Never recommend bulk approval. Blocked, source-unavailable, conflicting or unqualified proposals SHALL name required proof and state that yes cannot make them executable. Historical-maintenance proposals SHALL point to documented preparation and separate operation authorization; normal sync cannot execute them. Uncertain requests SHALL point to preserved journal/readback evidence and prohibit resend, journal deletion or journal edits as release instructions. False matches SHALL retain documented manual resolution without granting cleanup authority. + +Approved work awaiting the qualified per-run limit SHALL report remaining automatic work and its hold. Do not request approval again or force reruns. Simulation SHALL explain that approval executes no financial write in that cycle. Instructions SHALL NOT change qualification, approval, recovery, locking, financial effects or exits. Required actions SHALL join the one per-run notification; keys, paths and financial comparisons remain only in private logs/files. + +Baseline: docs/activity-review.md documents two sections and execution paths. The production warning names only the internal path and held-event count. + +#### Scenario: Distinct safe instructions from one shared file +- **WHEN** `python -m pytest -q tests/test_activity_review.py -k operator_action_review` creates normal-run, historical, blocked, invalid-choice, stale, source-unavailable, uncertain and approved-budget-deferred cases at a custom path +- **THEN** logs show path, section/key, reason and next action; event counts do not assume candidate counts; no bulk yes, fabricated host path, maintenance permission or uncertainty reset is suggested; financial requests and exits remain unchanged. diff --git a/openspec/specs/connector-runtime/spec.md b/openspec/specs/connector-runtime/spec.md index b4c70d7..94625f5 100644 --- a/openspec/specs/connector-runtime/spec.md +++ b/openspec/specs/connector-runtime/spec.md @@ -53,7 +53,7 @@ Check: `.venv/bin/python -m pytest -q tests/test_parsing.py tests/test_orchestra Absent or blank `DRY_RUN` SHALL simulate and emit WARNING `DRY_RUN not set: simulation, nothing written` on every invocation, including configuration failure. After trimming and lowercasing, `1/true/yes/on` SHALL simulate; only `0/false/no/off` SHALL request live writes. Any other value SHALL fail configuration with exit 1 before any HTTP call. After rejection, only the configured Apprise Failure notification SHALL be permitted; no IBKR/Ghostfolio HTTP calls or financial writes SHALL occur. Other safety guards SHALL still apply. -Simulation SHALL run login, reads and the existing preview pipeline without activity, cash or platform writes. Failed simulations SHALL send the configured Failure notification. Successful simulations, token-expiry reminders and previews SHALL send no notification unless corporate-action-detection requires a hold, changed finding or resumption summary. Corporate-action evidence and state SHALL be evaluated in simulation without financial writes. Missing/unwritable corporate-action state or an active hold SHALL exit 1 with the existing consolidated failure path; a clean simulation with valid state retains exit 0. Existing TZ-forced notifications SHALL remain for explicit live mode. +Simulation SHALL run login, reads and the existing preview pipeline without activity, cash or platform writes. Failed simulations SHALL send the configured Failure notification. Successful simulations SHALL notify for required user actions under Notifications, including required token renewal and timezone correction. A deliberate clean preview alone SHALL send no notification. Existing corporate-action findings and resumption notifications remain eligible. Corporate-action evidence and state SHALL be evaluated in simulation without financial writes. Missing/unwritable corporate-action state or an active hold SHALL exit 1 with the existing consolidated failure path; a clean simulation with valid state retains exit 0. Existing TZ-forced notifications SHALL remain for explicit live mode. README configuration, Docker/Compose examples, setup, 3.0 upgrade note and changelog SHALL explain this breaking rc.2 change and explicit false opt-in. Examples SHALL select simulation explicitly; instructions SHALL explain changing to false after review. @@ -75,6 +75,10 @@ Baseline replaced: README configuration; `test_load_config_dry_run_truthy_values - **WHEN** `python -m pytest -q tests/test_corporate_actions.py -k runtime_contract` runs the baseline and corporate-action variants for this requirement - **THEN** baseline behavior remains except for the specified safety holds, persistent-state prerequisites and consolidated corporate-action notification rules; no corporate-action financial request is sent. +#### Scenario: Required-action reporting compatibility +- **WHEN** `python -m pytest -q tests -k operator_action` exercises this requirement with synthetic inputs and mocked HTTP +- **THEN** required actions emit at least WARNING and join one configured notification; financial selection, payloads, account fences and exit codes remain unchanged. + ### Requirement: Flex selection and conversion The connector SHALL select Trade children excluding CASH/OPT, DETAIL dividend/PIL/withholding grouped by security/day for historical deduplication, then partitioned by source account and action for new writes, and BASE_SUMMARY cash; ignore other cash types/levels for ordinary import and ignore AssetSummary. Before related BUY/SELL or corrections, CorporateAction rows SHALL follow corporate-action-detection. Require the CorporateActions section: selected-empty means no rows observed in the report window; absent means hold that verified source account's BUY/SELL, without blocking independent cash solely for that absence. Setup documentation SHALL list the required section and observed fields. Do not require OpenPositions or another export format. @@ -133,7 +137,7 @@ Each action SHALL have one payment day, one contract, one currency and one exDat ### Requirement: Mapping and duplicate reconciliation -Mapping SHALL prefer ISIN mappings, then IBKR symbols. The existing matching rules below SHALL apply when ACTIVITY_REVIEW_FILE is disabled. When enabled, activity-correction-review SHALL first inspect complete supported source events, including existing identities, and hold events with pending candidates before ordinary selection. No blank approval or deleted review entry SHALL authorize a related import. If all former candidates are proven assigned to other identified events, release the hold and repeat ordinary selection under activity-correction-review. Missing or inactive targets SHALL remain held. Review SHALL NOT combine trade fills or dividend actions into one correction. Trade IDs SHALL suppress reimport. Manual matching SHALL use quantity and ±2 days, including summed fills. Ambiguous split fills SHALL only be summed within the same parsed calendar day, side, resolved ticker, and ISIN; nearby mismatches SHALL warn/skip. Dividends SHALL match by account/comment (including the old symbol comment), or, for existing active YAHOO dividends of the same account and symbol whose comment does not start with `dividend#` (manual or legacy rows), at ±3 days. Connector-comment rows (`dividend#…`) SHALL NOT enter the proximity index, so a distinct payment next to an imported one is imported. A proximity skip SHALL stay a skip. The comparison SHALL use the exact source gross and the existing row's quantity × unitPrice, normalized to a common major unit (minor-unit markets as in conversion) and rounded to its minor unit; an unsupported denomination SHALL be "not comparable". Equal amounts SHALL log INFO "amounts agree" with both fee values; equal amounts do not prove the same payment. Otherwise it SHALL log one WARNING naming both dates, both gross values with currencies (or "not comparable"), both fee values, and the operator action: check IBKR and add the payment by hand if it is distinct. It SHALL NOT change exit status or notifications. +Mapping SHALL prefer ISIN mappings, then IBKR symbols. The existing matching rules below SHALL apply when ACTIVITY_REVIEW_FILE is disabled. When enabled, activity-correction-review SHALL first inspect complete supported source events, including existing identities, and hold events with pending candidates before ordinary selection. No blank approval or deleted review entry SHALL authorize a related import. If all former candidates are proven assigned to other identified events, release the hold and repeat ordinary selection under activity-correction-review. Missing or inactive targets SHALL remain held. Review SHALL NOT combine trade fills or dividend actions into one correction. Trade IDs SHALL suppress reimport. Manual matching SHALL use quantity and ±2 days, including summed fills. Ambiguous split fills SHALL only be summed within the same parsed calendar day, side, resolved ticker, and ISIN; nearby mismatches SHALL warn/skip. Dividends SHALL match by account/comment (including the old symbol comment), or, for existing active YAHOO dividends of the same account and symbol whose comment does not start with `dividend#` (manual or legacy rows), at ±3 days. Connector-comment rows (`dividend#…`) SHALL NOT enter the proximity index, so a distinct payment next to an imported one is imported. A proximity skip SHALL stay a skip. The comparison SHALL use the exact source gross and the existing row's quantity × unitPrice, normalized to a common major unit (minor-unit markets as in conversion) and rounded to its minor unit; an unsupported denomination SHALL be "not comparable". Equal amounts SHALL log INFO "amounts agree" with both fee values; equal amounts do not prove the same payment. Otherwise it SHALL log one WARNING naming both dates, both gross values with currencies (or "not comparable"), both fee values, and the operator action: check IBKR and add the payment by hand if it is distinct. It SHALL NOT change exit status. Required manual checks SHALL join the consolidated notification under Notifications. #### Scenario: Existing entries - **WHEN** existing entries match @@ -163,6 +167,10 @@ Check: `.venv/bin/python -m pytest -q tests -k dividend_proximity_amount`. - **WHEN** `.venv/bin/python -m pytest -q tests/test_activity_review.py -k review_runtime_contract` exercises this requirement with review disabled, enabled and later disabled after a notes-bearing correction - **THEN** the disabled baseline remains valid, the explicit review exception follows activity-correction-review, and all supported historical reference generations remain readable. +#### Scenario: Required-action reporting compatibility +- **WHEN** `python -m pytest -q tests -k operator_action` exercises this requirement with synthetic inputs and mocked HTTP +- **THEN** required actions emit at least WARNING and join one configured notification; financial selection, payloads, account fences and exit codes remain unchanged. + ### Requirement: Position and acceptance guards Incomplete/redacted/invalid-profile listings SHALL stop startup; assetProfile SHALL override legacy SymbolProfile. Draft/excluded/non-YAHOO records SHALL not back reconciliation. Future candidates SHALL defer. Corporate-action identity and effective-quantity holds SHALL apply before related BUY/SELL dispatch, including optional corrections. Use proven dated native adjustments or already-normalized units exactly once for the position guard. Preserve stored activity values for identities and readback. Do not infer native-adjusted holdings by summing raw activities. Missing proof SHALL hold the bounded scope under corporate-action-detection; all existing acceptance and uncertainty fences SHALL remain. Release SHALL not authorize importing unmatched pre-event raw trades into an already-normalized history. Re-evaluate related conclusions after each confirmed review correction before further dispatch. Unknown effective timing SHALL retain the affected candidate hold. Sales SHALL pass net-quantity and accepted-buy checks; unknown acceptance SHALL block later same-target queries within this run. @@ -193,9 +201,31 @@ Evidence: README “How it works”; `python -m pytest -q tests/test_orchestrati ### Requirement: Notifications -Configured notifications SHALL send at most one consolidated Failure, corporate-action, legacy-expiry or forced-preview summary per completed run, without affecting exit status. Failure SHALL notify even for default or truthy simulation and invalid `DRY_RUN`. A failure with an expiry reminder or forced preview SHALL use one Failure summary, never a second dispatch. Evidence-maintenance warnings SHALL join the same consolidated dispatch without changing exit status or adding a trade hold when required proof remains valid and durable. Required-proof failures SHALL retain their existing fences. Without configured Apprise, no notification SHALL be sent. +Configured notifications SHALL send at most one consolidated summary per completed run. Failures, required user actions, existing corporate-action findings/resumptions, legacy-token expiry and forced-preview reasons SHALL share that dispatch. Failure SHALL retain title priority. Every observed required user action SHALL notify on each completed run until current evidence shows that it no longer requires action, including default, blank and truthy simulation. Approval, acknowledgment, previous delivery and unchanged findings SHALL NOT suppress a still-required action. A clean success SHALL send no success heartbeat. Recovered retries and informational findings that need no action SHALL NOT create a required-action notification. Existing corporate-action first/changed nonblocking findings and verified resumptions SHALL retain their own notification rules. + +Failure SHALL notify even with invalid DRY_RUN or configuration. Deliberate simulation alone SHALL not notify. Required token renewal or timezone correction SHALL notify in simulation too. Repeated observations of the same event/reason SHALL contribute once to the summary. Separate reasons SHALL remain visible without claiming their counts are disjoint. Required actions SHALL include safe plain-English next steps and fixed reason codes/counts. External summaries SHALL contain no account names/IDs, securities, activity/source IDs, financial values, local paths, hosts, credentials, destination URLs or raw errors. Private logs SHALL provide details and applicable paths. + +Without configured Apprise, no send SHALL occur. Each completed run SHALL log a notification result: `not required`, `disabled`, `invalid configuration`, `sent`, `failed`, or `timeout, delivery unconfirmed`. Required action with disabled/invalid delivery SHALL produce at least WARNING and a safe configuration instruction. Failed delivery SHALL also warn. Never claim ntfy received a message merely because notifications are configured. `sent` means the transport reported success, not device receipt. Retain fixed failure reasons without destination values. Do not notify recursively about delivery failure. + +Preserve one bounded worker attempt, its credential-isolated environment and its timeout. Never retry within the run after failure or timeout. Delivery outcome SHALL NOT change financial behavior or exit status. Actions still observed next run SHALL remain eligible regardless of delivery outcome. Corporate notification-state locking, revalidation and post-send confirmation SHALL remain. Delivery-state failure SHALL not change financial evidence or release a hold. Interruptions SHALL NOT claim completion or send a completion notification. + +Baseline replaced: tests/test_finalize.py success/warning silence and the 2026-10-27 production cycle with 68 review-held source events and no notification. Commands below specify future acceptance, not current passing evidence. + +#### Scenario: Repeated required action and resolution +- **WHEN** `python -m pytest -q tests/test_finalize.py -k operator_action_reminders` exercises live and all simulation modes across unchanged, changed and resolved conditions +- **THEN** each completed run with a required action makes one configured dispatch; ordinary deduplication does not hide the action; resolution ends that reminder without a success heartbeat. + +#### Scenario: One private summary across all reasons +- **WHEN** `python -m pytest -q tests/test_finalize.py -k operator_action_summary` combines several accounts, failures, review, diagnostics, mapping, token expiry, forced preview and corporate-action reasons with secret and financial canaries +- **THEN** one redacted dispatch includes safe instructions and fixed reason counts; failure retains priority; repeated observations do not inflate the same event/reason count. + +#### Scenario: Explicit delivery result +- **WHEN** `python -m pytest -q tests/test_finalize.py tests/test_notify.py -k operator_action_delivery` exercises clean silence, absent/empty/invalid configuration, success, worker failure, timeout and interruption +- **THEN** completed runs log the correct result; required action with disabled delivery warns; no destination leaks, resend or recursion occurs; interruption never claims completion. -Successful simulation, including default/blank/truthy `DRY_RUN`, SHALL send no success, expiry-only or preview notification. Corporate-action-detection SHALL add active holds each run, nonblocking first/changed findings and verified resumption to that same dispatch, including simulation. Corporate-action summaries SHALL retain the existing identifier/amount/secret redaction. Failed or unconfirmed delivery SHALL leave its finding eligible next run. Record delivery through separately locked, revalidated companion notification state after send; its failure SHALL not change financial decisions or exit status. With explicit false `DRY_RUN`, missing or invalid IANA `TZ` SHALL retain the forced-preview summary even when every account succeeds, and the summary SHALL name the reason. Success/warnings without expiry or a corporate-action notification reason, and interruptions, SHALL send nothing. +#### Scenario: Existing transport and financial contracts +- **WHEN** `python -m pytest -q tests/test_finalize.py tests/test_notify.py tests/test_token_expiry.py tests/test_corporate_actions.py` runs offline with synthetic inputs and mocked HTTP +- **THEN** worker isolation, bounded delivery, corporate state and financial fences remain valid with the explicitly changed action and simulation notification rules. #### Scenario: Consolidated outcome - **WHEN** `python3 -m pytest -q tests/test_finalize.py tests/test_token_expiry.py -k 'test_several_failed_accounts_make_one_dispatch_after_every_account_ran or test_summary_carries_no_secret_identifier_host_or_amount or test_notification_problems_keep_exit_code_and_account_processing or test_success_notifies_once_after_both_accounts_and_preserves_exit or test_reminder_is_run_local_and_multiple_runs_can_repeat_it'` runs with explicit live mode @@ -203,11 +233,11 @@ Successful simulation, including default/blank/truthy `DRY_RUN`, SHALL send no s #### Scenario: Forced preview notifies - **WHEN** `python3 -m pytest -q tests/test_finalize.py -k historical_diagnostic_preflight` runs with absent/invalid `TZ`, valid `TZ`, and default/truthy/explicit-false `DRY_RUN` -- **THEN** explicit false with invalid TZ sends one summary naming the forced-preview reason even on success; successful default/truthy simulations without a corporate-action notification reason stay silent; failures send Failure once. +- **THEN** explicit false with invalid TZ sends one summary naming the forced-preview reason even on success; successful default/truthy simulations without a required-action or corporate-action notification reason stay silent; failures send Failure once. #### Scenario: Notification compatibility - **WHEN** `python3 -m pytest -q tests/test_finalize.py -k dry_run_modes` exercises default, blank, all truthy values, explicit false and invalid input, with success, failure, expiry reminders, previews, TZ forcing, configured/disabled Apprise and delivery failure -- **THEN** failed runs send one Failure notification when configured, including default/truthy simulation and invalid input; successful simulations, expiry-only reminders and previews without a corporate-action notification reason send nothing; explicit false retains existing notification behavior; delivery never changes exit status. +- **THEN** failed runs send one Failure notification when configured, including default/truthy simulation and invalid input; successful simulations without required action or a corporate-action reason send nothing; expiry and preview follow the required-action rule in every mode; delivery never changes exit status. #### Scenario: Corporate-action runtime integration - **WHEN** `python -m pytest -q tests/test_corporate_actions.py -k runtime_contract` runs the baseline and corporate-action variants for this requirement @@ -239,7 +269,7 @@ With review disabled, processing SHALL extract cash identity without quantity ch ChangeInDividendAccruals SHALL be mandatory. Required attributes SHALL cover candidate payment/relevant accrual keys and `quantity/code/date/payDate/reportDate`. Nonempty key mismatches SHALL exclude unrelated accruals; missing keys SHALL NOT prove irrelevance. Unrelated rows SHALL NOT block candidates. Withholding SHALL retain cash evidence without accrual-schema requirements; new action writes SHALL require the explicit tax association in Flex selection and conversion. Missing section/relevant attributes SHALL block account dividends; present-but-empty evidence SHALL reject affected groups. Empty sections without payments SHALL be valid. -Rejections SHALL log distinct reasons, never mark rejected payments imported, and leave trades/cash operating. Partial v2 coverage, action-context conflicts and the new action-partition failures specified in Flex selection and conversion SHALL fail the account result; other existing local rejections SHALL NOT change success, exit, or notifications. Other failures SHALL retain current behavior. +Rejections SHALL log distinct reasons, never mark rejected payments imported, and leave trades/cash operating. Partial v2 coverage, action-context conflicts and the new action-partition failures specified in Flex selection and conversion SHALL fail the account result; other existing local rejections SHALL NOT change success or exit. A rejected payable event requiring investigation SHALL emit at least WARNING and join the required-action notification. Other failures SHALL retain current behavior. #### Scenario: Compatibility and isolation - **WHEN** `.venv/bin/python -m pytest -q tests -k dividend_structured_input` runs old, empty, short-window, unrelated, missing, conflicting and partial-v2 cases @@ -253,6 +283,10 @@ Rejections SHALL log distinct reasons, never mark rejected payments imported, an - **WHEN** `.venv/bin/python -m pytest -q tests/test_activity_review.py -k review_runtime_contract` exercises this requirement with review disabled, enabled and later disabled after a notes-bearing correction - **THEN** the disabled baseline remains valid, the explicit review exception follows activity-correction-review, and all supported historical reference generations remain readable. +#### Scenario: Required-action reporting compatibility +- **WHEN** `python -m pytest -q tests -k operator_action` exercises this requirement with synthetic inputs and mocked HTTP +- **THEN** required actions emit at least WARNING and join one configured notification; financial selection, payloads, account fences and exit codes remain unchanged. + ### Requirement: Forward dividend upgrade guidance README/setup SHALL require enrichment before upgrade, explain rejections/removed inference and backup/prior-image-digest rollback without financial restoration. Extra fields SHALL retain prior-runtime compatibility. @@ -297,14 +331,20 @@ Check: `.venv/bin/python -m pytest -v tests/test_dividend_diagnostics.py -k evid ### Requirement: Diagnostic execution and visibility -The diagnostic SHALL use existing reads, ENV and logs without persistence or financial writes. INFO SHALL summarize coverage/categories; anomaly/uncertainty details SHALL include reasons, activity IDs and source references with quantities/monetary deltas when supported. The per-account import line SHALL include the same `to check`, `informational` and `noise` counts as the dividend report, so its duplicate count cannot be mistaken for the report result. Logs remain private. Findings SHALL NOT alter exits/notifications; technical failures retain current behavior. Live runs SHALL NOT execute the diagnostic. README SHALL explain one-off Docker DRY_RUN without CRON, categories, limits and unchanged financial state. +Each diagnostic item SHALL declare either a user action with an observable resolution condition or an informational evidence limitation. An evidence limitation remains visible but does not trigger a reminder. + +The diagnostic SHALL use existing reads, ENV and logs without persistence or financial writes. INFO SHALL summarize coverage/categories. Items requiring user checks SHALL emit at least WARNING; anomaly/uncertainty details SHALL include reasons, activity IDs and source references with quantities/monetary deltas when supported. The per-account import line SHALL include the same `to check`, `informational` and `noise` counts as the dividend report, so its duplicate count cannot be mistaken for the report result. Logs remain private. Findings SHALL NOT alter exits. Items requiring user action SHALL join Notifications in simulation too. Information and noise alone SHALL not trigger a required-action notification; technical failures retain current behavior. Live runs SHALL NOT execute the diagnostic. README SHALL explain one-off Docker DRY_RUN without CRON, categories, limits and unchanged financial state. #### Scenario: Dry-run report - **WHEN** `.venv/bin/python -m pytest -v tests -k 'historical_diagnostic or coherent_dividend_import_summary'` runs mocked dry and live accounts -- **THEN** dry-run details and the import line expose identical report counts without writes; live behavior, exits and notifications remain unchanged. +- **THEN** dry-run details and the import line expose identical report counts without writes; live diagnostic selection, financial behavior and exits remain unchanged; required actions join the consolidated notification. Baseline: the 2026-10-15 real dry-run import line reported duplicate skips while the report separately reported items to check, without a link between the numbers. +#### Scenario: Required-action reporting compatibility +- **WHEN** `python -m pytest -q tests -k operator_action` exercises this requirement with synthetic inputs and mocked HTTP +- **THEN** required actions emit at least WARNING and join one configured notification; financial selection, payloads, account fences and exit codes remain unchanged. + ### Requirement: Timezone prerequisites and parser compatibility Before processing any account, missing or invalid IANA `TZ` SHALL force the entire run into `DRY_RUN`, prevent every Ghostfolio financial write, emit one WARNING line with the reason, and mention the forced preview in the final notification. @@ -429,7 +469,7 @@ The classes SHALL remain coherent, information, noise, outside window and to che A complete single legacy activity with equal quantity and fee SHALL be `legacy declared-rate rounding` information when its Ghostfolio gross equals quantity × the matched payment `Re` `grossRate` and the paid cash gross differs by exactly one minor unit. Its reason SHALL state that the legacy row uses the declared rate and IBKR paid a rounded cash amount. Its sole action SHALL read `No action: IBKR rounding accepted.` The block SHALL stay visible at INFO as audit information. It SHALL NOT suggest that IBKR revised the amount after import. A one-minor-unit fee difference, or any one-minor-unit gross difference outside this proven pattern, SHALL remain to check as `minimal difference, cause unknown`. No finding SHALL automatically correct an activity. -After informational findings, one INFO line SHALL read `N to check, N informational, N noise`, followed by each item to check. Findings SHALL NOT change exits or notifications. +After informational findings, one INFO line SHALL read `N to check, N informational, N noise`, followed by each item to check at WARNING or above. Findings SHALL NOT change exits. Items to check SHALL join the required-action notification. Information and noise alone SHALL not trigger it. #### Scenario: Plain, quantized, single-action report - **WHEN** `.venv/bin/python -m pytest -v tests/test_dividend_diagnostics.py tests/test_check_dividend_report.py -k 'coherent_dividend_format or check_dividend_report'` formats decimals, unavailable values, empty/nonempty summaries and blocks with match proposals @@ -457,6 +497,10 @@ After informational findings, one INFO line SHALL read `N to check, N informatio Baseline: `scripts/check_dividend_report.py` currently detects Python representations, float noise, contradictory actions, duplicate rows and false unsupported-currency reasons in the saved 2026-10-15 report. +#### Scenario: Required-action reporting compatibility +- **WHEN** `python -m pytest -q tests -k operator_action` exercises this requirement with synthetic inputs and mocked HTTP +- **THEN** required actions emit at least WARNING and join one configured notification; financial selection, payloads, account fences and exit codes remain unchanged. + ### Requirement: Cash target guard Configuration SHALL refuse a `GHOST_ACCOUNT_NAMES` list where two effective targets, after trimming, are the same; when `GHOST_ACCOUNT_NAMES` is empty the targets are the IBKR account ids. The error SHALL name the duplicated target, and the run SHALL stop with a non-zero exit before any network call or write. Before a cash write, the IBKR base currency (the `AccountInformation` `currency` attribute) SHALL equal the `currency` of the Ghostfolio account returned by the existing account GET. On a difference the connector SHALL log an error naming both currencies, send no cash PUT, and mark the account failed. An absent base currency SHALL log an error naming the missing Flex field, send no cash PUT, and mark the account failed. Matching currencies SHALL behave as before. Under `DRY_RUN` the GET and the check SHALL still run; no PUT is sent. @@ -481,7 +525,7 @@ Configuration SHALL refuse a `GHOST_ACCOUNT_NAMES` list where two effective targ Under DRY_RUN, every dividend item to check that has a candidate SHALL carry one `Proposed match` section. The section SHALL show the IBKR dividend and candidate Ghostfolio rows side by side, each row labelled `Keep`, `Keep, quantity differs`, `Duplicate` or `Differs`. Confidence SHALL be strong for the same security, local day, gross and fee, or weak for a candidate without equal gross and fee. A strong row selected to keep whose quantity differs from IBKR SHALL read `Keep, quantity differs`, never plain `Keep`. Quantity SHALL have no tolerance. With no candidate the section SHALL read `no confident match`. Weak proposals SHALL label no row `Keep` or `Keep, quantity differs`. -The proposed-match section SHALL NOT contain an action line. The enclosing block's single `Action:` line SHALL name the row and operation, or state that no action is needed. When the reason compares a Ghostfolio activity, the Action SHALL name that same activity, shown on the block's `Ghostfolio: activity ...` line, never another candidate. A quantity-difference reason SHALL request a manual quantity comparison for that activity and SHALL NOT state that no action is needed. `scripts/check_dividend_report.py` SHALL report a violation if the Action targets another candidate instead of the compared activity. Historical rows that split the same gross SHALL remain visible as `Differs`; their cleanup is out of scope and remains a separate reversible-maintenance idea. A legacy symbol-comment row that owns the source group SHALL appear once as the compared row; it SHALL NOT receive a second legacy-twin block. The proposal SHALL NOT write, repair or change import selection, classes, categories, summary counts, exits or notifications. Text SHALL follow the report's plain and quantized display rules. +The proposed-match section SHALL NOT contain an action line. The enclosing block's single `Action:` line SHALL name the row and operation, or state that no action is needed. When the reason compares a Ghostfolio activity, the Action SHALL name that same activity, shown on the block's `Ghostfolio: activity ...` line, never another candidate. A quantity-difference reason SHALL request a manual quantity comparison for that activity and SHALL NOT state that no action is needed. `scripts/check_dividend_report.py` SHALL report a violation if the Action targets another candidate instead of the compared activity. Historical rows that split the same gross SHALL remain visible as `Differs`; their cleanup is out of scope and remains a separate reversible-maintenance idea. A legacy symbol-comment row that owns the source group SHALL appear once as the compared row; it SHALL NOT receive a second legacy-twin block. The proposal SHALL NOT write, repair or change import selection, classes, categories, summary counts or exits. Its items to check SHALL join the required-action notification. Text SHALL follow the report's plain and quantized display rules. #### Scenario: Strong match with a legacy twin - **WHEN** `.venv/bin/python -m pytest -v tests/test_dividend_diagnostics.py -k 'dividend_match_strong or coherent_dividend_action'` formats a canonical row with a separate free-text legacy twin @@ -497,7 +541,7 @@ The proposed-match section SHALL NOT contain an action line. The enclosing block #### Scenario: Display only - **WHEN** `.venv/bin/python -m pytest -v tests/test_dividend_diagnostics.py -k 'dividend_match_unchanged or coherent_dividend_format'` compares findings before and after formatting -- **THEN** classes, categories, summary counts, exits and notifications are unchanged; displayed amounts are quantized and plain. +- **THEN** classes, categories, summary counts and exits are unchanged; displayed amounts are quantized and plain; formatting SHALL not add a notification for an already counted action. #### Scenario: Real read-only acceptance - **WHEN** the operator-authorized real dry-run log is read after `python3 scripts/check_dividend_report.py ` exits 0 @@ -505,13 +549,21 @@ The proposed-match section SHALL NOT contain an action line. The enclosing block Baseline: current proposed-match tests preserve classification. The saved 2026-10-15 report also shows a compared connector row with quantity 1 versus IBKR 100, equal gross/fee and two historical rows splitting that gross: the proposal labels the connector row `Keep`, while the Action targets a historical `Differs` row. This is the gap addressed by task 3.2. +#### Scenario: Required-action reporting compatibility +- **WHEN** `python -m pytest -q tests -k operator_action` exercises this requirement with synthetic inputs and mocked HTTP +- **THEN** required actions emit at least WARNING and join one configured notification; financial selection, payloads, account fences and exit codes remain unchanged. + ### Requirement: Minimal native lab evidence -The connector SHALL emit only three new evidence kinds through `log.debug`: `window`, `preview` and `cash`. Each message SHALL be `LAB_EVIDENCE ` followed by compact JSON with exactly `kind` and `value`. The normal timestamp and DEBUG logging prefix SHALL precede the message. JSON escaping SHALL keep each message on one physical line. JSON numbers SHALL be finite. Unserializable or nonfinite evidence SHALL be omitted without changing existing operations. At INFO, output SHALL be byte-identical to the pre-change implementation. Existing behavior, guards, requests and notifications SHALL remain unchanged. Evidence SHALL NOT dump secrets, URLs, XML, ENV, configuration, headers or response bodies. README SHALL state that lab acceptance needs `LOG_LEVEL=DEBUG`. +The connector SHALL emit only three new evidence kinds through `log.debug`: `window`, `preview` and `cash`. Each message SHALL be `LAB_EVIDENCE ` followed by compact JSON with exactly `kind` and `value`. The normal timestamp and DEBUG logging prefix SHALL precede the message. JSON escaping SHALL keep each message on one physical line. JSON numbers SHALL be finite. Unserializable or nonfinite evidence SHALL be omitted without changing existing operations. The evidence addition SHALL emit no LAB_EVIDENCE at INFO. Operational logging and required-action reporting SHALL govern new framing and summaries; the former global byte-identical INFO constraint SHALL not prohibit those changes. Evidence payloads, financial behavior, guards and requests SHALL remain unchanged. Evidence SHALL NOT dump secrets, URLs, XML, ENV, configuration, headers or response bodies. README SHALL state that lab acceptance needs `LOG_LEVEL=DEBUG`. #### Scenario: Framing and INFO compatibility - **WHEN** `.venv/bin/python -m pytest -q tests/test_run_evidence.py -k 'format or info or nonfinite'` runs with synthetic inputs and mocked HTTP -- **THEN** DEBUG evidence parses on single physical lines, preserves escaped payload text and finite numbers, excludes configuration secrets, and adds no HTTP calls; formatted INFO bytes equal the pre-change output, including the per-activity preview list. +- **THEN** DEBUG evidence parses on single physical lines, preserves escaped payload text and finite numbers, excludes configuration secrets, and adds no HTTP calls; INFO contains no LAB_EVIDENCE; per-activity preview message content remains unchanged while framing, levels and operational summaries follow the reporting requirements. + +#### Scenario: Required-action reporting compatibility +- **WHEN** `python -m pytest -q tests -k operator_action` exercises this requirement with synthetic inputs and mocked HTTP +- **THEN** required actions emit at least WARNING and join one configured notification; financial selection, payloads, account fences and exit codes remain unchanged. ### Requirement: Flex window evidence @@ -760,3 +812,63 @@ Historical apply, inspect and rollback SHALL compare preserved activity business #### Scenario: Metadata and business drift - **WHEN** `python -m pytest -q tests/test_identity_migration.py -k preservation` simulates comment changes, server timestamps, derived values, exact numeric normalization, unknown fields and financial drift - **THEN** metadata alone does not fail readback, business or unknown-field drift refuses further writes, raw tamper evidence remains checked, and reverse execution applies the same preservation rule. + +### Requirement: Required user actions + +Every condition requiring user intervention SHALL emit a private log at WARNING or above and enter the run's required-action summary. Classification SHALL follow the decision and evidence, not log-text matching or WARNING count. Each action SHALL identify its reason, affected scope, impact, safe next step and applicable configured file or setting. If there is no relevant file, name the service or UI to inspect. Retain existing credential and source-account redaction rules. Reporting SHALL NOT add a financial request, source request, approval, retry or permission. + +The following inventory SHALL be covered without changing conversion, diagnostic categories, import selection or exits: + +| Condition | Required action | +|---|---| +| Review needing a choice, invalid approval, stale choice | Inspect the proposal and execution path; approve only if eligible and intended | +| Missing/conflicting review evidence, unqualified correction, historical maintenance | Obtain the stated proof or follow separately authorized maintenance; yes alone cannot release it | +| Uncertain financial request | Inspect preserved journal and destination proof; do not resend, delete or edit the journal | +| Corporate-action hold or required evidence/state missing | Inspect conclusions and recover proof; acknowledgment is not release | +| Unverified symbol fallback or different-symbol holding conflict | Verify the destination security or correct the configured mapping | +| Manual-trade proximity needing verification or reference repair | Compare source and destination before any manual edit | +| Dividend proximity discrepancy or diagnostic item to check | Inspect the comparison; amount agreement does not authorize cleanup | +| Signed tax or reversal with zero-net/negative evidence, orphan withholding, or rejected payable event | Inspect the preserved source and destination evidence; do not invent or resend a transaction | +| Evidence-limited diagnostic representation | Read the named report limitation and its impact; no user correction is requested until a concrete source or destination action exists | +| Nonblocking corporate evidence-maintenance finding | Inspect or restore the named evidence/state file; automatic retry remains distinct from a user intervention | +| Payable source event omitted for missing, invalid or conflicting evidence | Inspect the rejection and required source fields; never invent payment or quantity | +| Missing BASE_SUMMARY | Inspect report cash configuration; disclose that cash was not updated | +| Invalid configuration, authentication, source acquisition or unsuccessful financial processing | Inspect the stated reason and configuration/service; preserve recovery fences | +| Required token renewal or timezone correction | Correct the named setting before affected live processing | +| Required notification cannot be sent | Configure or repair Apprise; use the local log without recursive notification | + +Approved work awaiting a later qualified cycle SHALL be reported as deferred automatic work, not a new approval request. Future-dated events, normal exclusions, accepted rounding information, deliberate simulation and recovered retries SHALL not create a required action by themselves. Representation limits SHALL explain their impact without asking for an impossible correction. An informational limitation SHALL not enter the required-action summary unless the implementation names a concrete user action and an observable resolution condition. A required action SHALL name the evidence that proves resolution; a repeated diagnostic limitation SHALL remain visible as information when no such action exists. + +Re-evaluate relevant action evidence from the acquired reports and existing destination/review state, including events already skipped as duplicates. An unresolved mapping or review action SHALL not vanish merely because an import completed earlier. Add no persistent alert store or broker request. Missing current evidence SHALL NOT be described as resolution. Retained review/state findings SHALL keep their visibility. State coverage limits without certifying unobserved lifetime history. + +Baseline: app.py, sync.py, activity_review.py and diagnostics.py already emit operator instructions; tests/test_finalize.py treats ordinary successful warnings as silent. Commands specify future acceptance. + +#### Scenario: Decision inventory and unchanged financial behavior +- **WHEN** `python -m pytest -q tests -k operator_action_inventory` exercises the listed conditions, duplicate observations, two accounts, later deduplicated runs, recovered retries and deferred approved work +- **THEN** every required action has a reason and safe instruction at WARNING or above; only actual required actions enter repeated reminders; coverage limits remain explicit; original requests, payloads, approvals, fences and exits remain unchanged. + +### Requirement: Final operational result + +Every completed cycle SHALL distinguish `success`, `success with user action required`, and `failure`, independently of the unchanged exit code. A cycle with only automatic deferred work or an evidence limitation that has no user-resolvable action SHALL remain `success` and SHALL report that fact separately. Failure SHALL take priority and retain its action details. The result SHALL use ERROR for failure, WARNING for action-only success and INFO for clean success. Separately report proven accepted imports or simulated proposals, duplicate skips, unique review-held source events, other justified holds, deferred automatic work and cash outcomes per account. Unknown counts SHALL read `unavailable`, never zero. Label overlapping decisions; do not add them into a false reconciliation total. Proposal count SHALL not stand for held-event count. + +Cash outcomes SHALL distinguish simulated, request reported successful, not updated and uncertain/failed outcomes, using existing evidence. A successful request SHALL NOT be described as independent balance readback. No new financial readback is introduced. Cash success SHALL not imply held activities were imported. Show notification outcome alongside the cycle result without claiming device delivery. Early failures SHALL report unavailable fields honestly. Exits and financial processing SHALL remain unchanged. + +Baseline replaced: app.py emits `Sync complete` for pending-review holds. The 2026-10-27 production log separately reports 68 held events, zero new imports and cash success. + +#### Scenario: Held activities with independent cash +- **WHEN** `python -m pytest -q tests/test_orchestration.py tests/test_finalize.py -k operator_action_result` exercises review holds with successful cash, simulation, missing cash, uncertain writes and early configuration failure +- **THEN** status and independent outcomes match the evidence, never claim readback or complete import, include delivery status, and preserve exits and financial requests. + +### Requirement: Clear application log framing + +One-off and scheduled application logs SHALL use the same readable application timestamp, level and message. Scheduled output SHALL NOT be embedded in a scheduler message with another timestamp, level or channel. Application DEBUG/INFO SHALL use stdout; WARNING and above SHALL use stderr. No record SHALL appear on both streams. Scheduler startup, cycle start/success/failure and shutdown records SHALL remain available and distinguishable. Preserve numeric cron validation, child exit handling, signal handling, LOG_LEVEL filtering and secret suppression. A transport stream SHALL not be presented as application severity. + +Preserve LAB_EVIDENCE kind/value JSON, one-line escaping and DEBUG visibility. Logging changes SHALL NOT change financial requests or expose broker query tokens. Add no logger dependency, mandatory ENV or Docker socket access. + +#### Scenario: Direct and scheduled application levels +- **WHEN** `python -m pytest -q tests/test_entrypoint.py tests/test_run_evidence.py tests/test_logging.py -k operator_action_logging` runs isolated synthetic entrypoint and subprocess cases at each level and checks the scheduler invocation +- **THEN** records retain their level and intended stream without nested framing or duplicates; cron and lifecycle semantics remain; LAB_EVIDENCE parses unchanged. + +#### Scenario: User can locate and act on a finding +- **WHEN** `rg -n -A 12 'User action required|Notification result|container path' docs/configuration.md docs/activity-review.md README.md` runs after implementation +- **THEN** examples explain action-only success, repeats and clean silence, severity/streams, delivery states, volume resolution, safe review distinctions and unchanged financial authority, without private deployment paths. diff --git a/openspec/specs/corporate-action-detection/spec.md b/openspec/specs/corporate-action-detection/spec.md index 8418952..731e197 100644 --- a/openspec/specs/corporate-action-detection/spec.md +++ b/openspec/specs/corporate-action-detection/spec.md @@ -83,14 +83,22 @@ Retain known unresolved events across shorter report windows. Reconcile current ### Requirement: Consolidated corporate-action reporting -With Apprise configured, active corporate-action holds SHALL notify once in each completed run. Nonblocking findings SHALL notify on first appearance and material change. A previously held scope SHALL notify when fresh evidence permits resumption. Acknowledgment SHALL not suppress active holds. Simulation SHALL report these same observed conditions without claiming a financial execution. Coalesce these reasons with existing failure/expiry/forced-preview reasons into at most one dispatch per run. Unchanged nonblocking findings SHALL remain in the detailed report without another notification. +Maintenance findings SHALL state whether the connector will retry automatically or whether the operator must restore evidence/state. Only the latter is a repeated required-action reminder. -Notification bodies SHALL contain only safe fixed reason codes and counts. No account, security, activity ID, amount, host or credential SHALL appear. Notification failure SHALL not change financial decisions or exit status. Record successful notification fingerprints only after confirmed delivery. Finalization SHALL reacquire the companion notification lock, revalidate its generation and atomically record only the exact findings that were sent. A concurrent change SHALL not be overwritten. Failure to record delivery SHALL leave the findings eligible next run. A send timeout SHALL remain unconfirmed even if it may later arrive; do not retry during that run. Failed delivery SHALL remain eligible next run. Notification fingerprints SHALL not release or create financial holds, consume financial approvals, or suppress active-hold notifications. A crash between send and persistence can duplicate a later notification; no exactly-once claim is made. Active holds and hold-critical state failures SHALL degrade the run; nonblocking historical discrepancies alone SHALL not. +With Apprise configured, active corporate-action holds SHALL notify once in each completed run. Nonblocking findings that require no user action SHALL notify on first appearance and material change. A nonblocking finding requiring user action SHALL instead join the required-action reminder on every completed run until resolved, including simulation. A previously held scope SHALL notify when fresh evidence permits resumption. Acknowledgment SHALL not suppress active holds. Simulation SHALL report these same observed conditions without claiming a financial execution. Coalesce these reasons with existing failure/expiry/forced-preview reasons into at most one dispatch per run. Unchanged nonblocking findings that require no user action SHALL remain in the detailed report without another notification. Companion delivery fingerprints SHALL not suppress required-action reminders. The one summary SHALL also include review, mapping, diagnostic and other required-action reasons under connector-runtime. + +Notification bodies SHALL contain only safe fixed reason codes, counts and fixed plain-English instructions. No account, security, activity ID, amount, host or credential SHALL appear. Notification failure SHALL not change financial decisions or exit status. Record successful notification fingerprints only after confirmed delivery. Finalization SHALL reacquire the companion notification lock, revalidate its generation and atomically record only the exact findings that were sent. A concurrent change SHALL not be overwritten. Failure to record delivery SHALL leave the findings eligible next run. A send timeout SHALL remain unconfirmed even if it may later arrive; do not retry during that run. Failed delivery SHALL remain eligible next run. Notification fingerprints SHALL not release or create financial holds, consume financial approvals, or suppress active-hold notifications. A crash between send and persistence can duplicate a later notification; no exactly-once claim is made. Active holds and hold-critical state failures SHALL degrade the run; nonblocking historical discrepancies alone SHALL not. #### Scenario: Repeated notifications and resumption - **WHEN** `python -m pytest -q tests/test_corporate_actions.py -k notifications` runs first, unchanged, changed, held, resolved, simulated, failed-delivery, timeout, post-send crash, concurrent finalization, unavailable companion state and Apprise-disabled cases alongside other run failures - **THEN** one redacted summary follows the specified schedule, detailed findings appear every run, and notification delivery neither releases a hold nor changes the run outcome. +Each action-required corporate finding SHALL log at least WARNING with its reason, affected scope, configured state/evidence path when applicable and safe next step. Explain which proof is missing and which operations remain held. Do not present acknowledgment, state reset or journal edits as release. Private detail and public fixed-code/count redaction SHALL retain their separate roles. Existing trade holds, cash independence, exits and evidence retention SHALL remain unchanged. + +#### Scenario: Corporate and other required actions share delivery +- **WHEN** `python -m pytest -q tests/test_corporate_actions.py tests/test_finalize.py -k operator_action_corporate` exercises unchanged action-required findings, unchanged non-action findings, review actions, delivery failure, fingerprint loss, resolution and simulation +- **THEN** required actions repeat each completed run in one redacted dispatch, non-action findings retain first/change/resumption behavior, and notification state never changes financial decisions or creates another dispatch. + ### Requirement: Automatic report rotation and safety evidence retention The connector SHALL retain ordinary acquired reports for a fixed 30 days, measured as 720 hours from durable UTC acquisition time. On the next normal run after expiration, it SHALL automatically retire reports that are not required safety evidence. This SHALL also apply in simulation: local evidence maintenance is allowed, financial writes remain forbidden. There SHALL be no new retention ENV setting, byte quota, object-count ceiling or separate maintenance command. A period without executions does not run background cleanup. Re-reading an existing acquisition or changing its file timestamps SHALL NOT renew its age. Invalid, missing or future acquisition timestamps SHALL NOT justify early removal. Existing evidence without trustworthy acquisition time SHALL receive a single durable migration observation time and the full 30-day grace period. diff --git a/scripts/check_container_runtime.sh b/scripts/check_container_runtime.sh index cac41dd..ab142d5 100644 --- a/scripts/check_container_runtime.sh +++ b/scripts/check_container_runtime.sh @@ -112,6 +112,58 @@ echo "$scheduled_logs" grep -q 'Starting IBKR to Ghostfolio sync' <<< "$scheduled_logs" grep -q 'python -m ghostfolio_ibkr_sync' <<< "$scheduled_logs" +# Keep actual scheduled application streams separate. Supercronic lifecycle +# records can share a stream, but must not wrap application timestamp/level. +docker logs "$scheduled_id" > "$mapping_dir/scheduled.stdout" 2> "$mapping_dir/scheduled.stderr" +python3 - "$mapping_dir/scheduled.stdout" "$mapping_dir/scheduled.stderr" <<'PY_STREAMS' +import re +import sys +from pathlib import Path +stdout, stderr = [Path(name).read_text() for name in sys.argv[1:]] +pattern = re.compile(r"^\d{4}-\d\d-\d\d \d\d:\d\d:\d\d \[(DEBUG|INFO|WARNING|ERROR|CRITICAL)\] ") +assert stdout.count("Starting IBKR to Ghostfolio sync") == 1, stdout +assert "Starting IBKR to Ghostfolio sync" not in stderr +assert "Missing required environment variables: IBKR_TOKEN" in stderr +assert "Missing required environment variables: IBKR_TOKEN" not in stdout +for stream, allowed in ((stdout, {"DEBUG", "INFO"}), (stderr, {"WARNING", "ERROR", "CRITICAL"})): + for line in stream.splitlines(): + if match := pattern.match(line): + assert match[1] in allowed, line + assert not ("channel=" in line and re.search(r"\[(DEBUG|INFO|WARNING|ERROR)\]", line)), line +print("Scheduled application streams and native framing passed") +PY_STREAMS + +# Exercise the installed CLI with synthetic work at every level. No network, +# credentials, financial action or alternate logging implementation is used. +docker run --rm "${real_options[@]}" --entrypoint python "$image" -c ' +import logging +from ghostfolio_ibkr_sync import __main__ as cli +from ghostfolio_ibkr_sync.ibkr import log_lab_evidence +import os +os.environ["LOG_LEVEL"] = "DEBUG" +def synthetic(): + for level in (10, 20, 30, 40, 50): + logging.getLogger("probe").log(level, "STREAM_PROBE_%s", logging.getLevelName(level)) + log_lab_evidence("window", {"escaped": "one\ntwo"}) + return 0 +cli.main = synthetic +raise SystemExit(cli.run()) +' > "$mapping_dir/direct.stdout" 2> "$mapping_dir/direct.stderr" +python3 - "$mapping_dir/direct.stdout" "$mapping_dir/direct.stderr" <<'PY_DIRECT' +import json +import sys +from pathlib import Path +stdout, stderr = [Path(name).read_text() for name in sys.argv[1:]] +for name in ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"): + selected, other = (stdout, stderr) if name in ("DEBUG", "INFO") else (stderr, stdout) + assert selected.count("STREAM_PROBE_" + name) == 1 + assert "STREAM_PROBE_" + name not in other +lines = [line for line in stdout.splitlines() if "LAB_EVIDENCE " in line] +assert len(lines) == 1 and "[DEBUG] LAB_EVIDENCE " in lines[0] +assert json.loads(lines[0].split("LAB_EVIDENCE ", 1)[1]) == {"kind": "window", "value": {"escaped": "one\ntwo"}} +print("Installed CLI levels and LAB_EVIDENCE streams passed") +PY_DIRECT + # Notification worker with its production argv, from a directory that holds no # source: -I leaves the current directory and PYTHON* variables out of module # lookup, so only the installed package can answer. Malformed stdin must exit 10 diff --git a/tests/test_activity_review.py b/tests/test_activity_review.py index 043b008..3e688c3 100644 --- a/tests/test_activity_review.py +++ b/tests/test_activity_review.py @@ -240,7 +240,7 @@ def put(self, url, **kwargs): return response -def run_review(tmp_path, monkeypatch, *, outcome="success", approval="yes", dry_run=False, rows=None): +def run_review(tmp_path, monkeypatch, *, outcome="success", approval="yes", dry_run=False, rows=None, run_outcome=None): path = tmp_path / "review.yaml" server = ReviewServer(monkeypatch, rows, outcome) proposals = preview(rows) @@ -250,7 +250,7 @@ def run_review(tmp_path, monkeypatch, *, outcome="success", approval="yes", dry_ path.chmod(0o600) state = review.open_review(str(path), []) config = {"ghost_host": "http://ghost", "ghost_token": "secret", "dry_run": dry_run, - "activity_review_file": str(path)} + "activity_review_file": str(path), "_corporate_outcome": run_outcome} # This is an isolated execution-budget fixture, not image qualification. monkeypatch.setattr(review, "QUALIFIED_EXECUTION", ("synthetic-image", 2, 0.5)) blocked, held = review.process_review(state, config, source_events(), {"U1": "acc"}) @@ -1220,3 +1220,49 @@ def recalculate(self, url, **kwargs): assert not blocked and not held and len(server.writes) == 1 finally: review.close_review(state) + + +@pytest.mark.parametrize("approval,expected", [(None, "review_approval"), (True, "review_approval"), ("YES", "review_approval")]) +def test_operator_action_review_pending_private_path(tmp_path, monkeypatch, caplog, approval, expected): + from ghostfolio_ibkr_sync.outcome import new_run_outcome + out = new_run_outcome() + server, state, config, blocked, held = run_review(tmp_path, monkeypatch, approval=approval, run_outcome=out) + try: + assert out["actions"][expected] == held + assert str(state["path"]) in caplog.text and "activity_corrections" in caplog.text + assert "only lowercase literal yes" in caplog.text + assert not server.writes and not blocked + finally: + review.close_review(state) + + +@pytest.mark.parametrize("case,expected", [("unqualified", "review_evidence"), ("historical", "review_maintenance"), + ("blocked", "review_evidence"), ("uncertain", "review_uncertain"), + ("source unavailable", "review_evidence"), ("deferred", None)]) +def test_operator_action_review_distinct_paths(tmp_path, monkeypatch, caplog, case, expected): + from ghostfolio_ibkr_sync.outcome import new_run_outcome + out = new_run_outcome() + proposal = next(iter(preview().values())) + proposal["approve"] = "yes" + if case == "historical": + proposal["execution path"] = "historical maintenance" + if case == "blocked": + proposal.update(actionable=False, status="missing proof") + if case == "source unavailable": + proposal["status"] = case + monkeypatch.setattr(review, "QUALIFIED_EXECUTION", None if case == "unqualified" else ("synthetic", 2, 0)) + state = review.open_review(str(tmp_path / "custom.yaml"), []) + config = {"_corporate_outcome": out, "dry_run": True} + try: + review.report_review_outcome(state, config, {proposal["key"]: proposal}, {proposal["source event"]}, + {proposal["source event"]} if case == "uncertain" else set()) + assert set(out["actions"]) == ({expected} if expected else set()) + assert out["review_held"] == {proposal["source event"]} + if case == "uncertain": + assert "Do not resend, delete or edit the journal" in caplog.text + if case == "deferred": + assert out["deferred"] == {proposal["source event"]} + else: + assert str(state["path"]) in caplog.text + finally: + review.close_review(state) diff --git a/tests/test_auth.py b/tests/test_auth.py index 0396efc..7b6d03b 100644 --- a/tests/test_auth.py +++ b/tests/test_auth.py @@ -358,7 +358,9 @@ def get(url, headers=None, **kw): assert writes[1][2]["platformId"] == "synthetic-platform" else: assert writes == [] - assert len(sent) == int(failed) + assert len(sent) == 1 if failed: assert sent[0][0] == notify.NOTIFY_TITLE + else: + assert "dividend_rejected" in sent[0][1] assert reads and all(auth == "Bearer fresh-bearer" for _, auth in reads) diff --git a/tests/test_conversion.py b/tests/test_conversion.py index 9275491..1b67fda 100644 --- a/tests/test_conversion.py +++ b/tests/test_conversion.py @@ -293,3 +293,25 @@ def test_v2_source_qualification_partition_cancellation_refund_and_pil(reverse_c # Opposite signed tax conserves before clamping; separate clamps differ. assert sum(a["fee"] for a in activities) == 2 assert sync.convert_dividend_to_activity(group, "dest", {}, {}) is None # unequal quantities + + +@pytest.mark.parametrize("case,reason", [("quantity", "dividend_rejected"), ("reversal", "dividend_reversal"), + ("tax", "dividend_signed_tax"), ("zero", None)]) +def test_operator_action_inventory_dividend_local_rejections(case, reason): + from ghostfolio_ibkr_sync.outcome import new_run_outcome + out = new_run_outcome() + row = div(quantity="bad" if case == "quantity" else "100", amount=-25 if case == "reversal" else 0 if case in ("tax", "zero") else 25, + tax=-3 if case == "tax" else 0) + assert sync.convert_dividend_to_activity(row, "acc", {ISIN: "KO"}, {}, outcome=out) is None + assert set(out["actions"]) == ({reason} if reason else set()) + + +def test_operator_action_inventory_dividend_reason_mapping_complete(): + import ast + from pathlib import Path + tree = ast.parse(Path(sync.__file__).read_text()) + reasons = {node.args[0].value for node in ast.walk(tree) if isinstance(node, ast.Call) + and isinstance(node.func, ast.Name) and node.func.id == "reject" and node.args + and isinstance(node.args[0], ast.Constant)} + assert reasons <= set(sync.DIVIDEND_REJECTION_ACTIONS) + assert all(sync.DIVIDEND_REJECTION_ACTIONS[reason] == "dividend_rejected" for reason in reasons) diff --git a/tests/test_corporate_actions.py b/tests/test_corporate_actions.py index d7a4e67..73a775a 100644 --- a/tests/test_corporate_actions.py +++ b/tests/test_corporate_actions.py @@ -513,7 +513,7 @@ def denied(file, *args, **kwargs): "_corporate_store": store, "_corporate_outcome": outcome} sent = [] monkeypatch.setenv("DRY_RUN", "true" if simulation else "false") - monkeypatch.setattr(app, "load_notification_config", lambda: {} if configured else None) + monkeypatch.setattr(app, "load_notification_delivery", lambda: ({}, "configured") if configured else (None, "disabled")) monkeypatch.setattr(app, "send_notification", lambda *args: sent.append(args) or "sent") app.refresh_corporate_decisions(config, {}, {"activities": [], "count": 0}, {"qty": defaultdict(float)}) assert not config["_corporate_decisions"]["SOURCE"]["account_held"] @@ -1639,3 +1639,26 @@ def test_holds_distinct_sources_cannot_certify_shared_destination_ownership( assert positions["qty"][("TARGET", "SAMPLE.T")] == 100 finally: ca.close_store(store) + + +@pytest.mark.parametrize("status", ["sent", "failed", "timeout"]) +def test_operator_action_corporate_repeats_with_review_and_fingerprint_loss(monkeypatch, tmp_path, status): + from ghostfolio_ibkr_sync import app, corporate_actions as ca + from ghostfolio_ibkr_sync.outcome import new_run_outcome, record_action + tmp_path.chmod(0o700) + monkeypatch.setenv("APPRISE_URLS", '["json://notify.invalid/hook"]') + monkeypatch.setenv("DRY_RUN", "true") + sent = [] + monkeypatch.setattr(app, "send_notification", lambda cfg, title, body: sent.append(body) or status) + filename = str(tmp_path / "state.yaml") + decision = notification_decision(False) + decision["findings"]["EVENT1"]["conclusions"]["history"] = "bounded_cost_dates_differ" + for _ in range(2): + out = new_run_outcome() + ca.prepare_notifications(out, filename, {"PRIVATE": decision}) + record_action(out, "review_approval", "PRIVATE-REVIEW") + app.finalize_run(out, 0) + assert len(out["actions"]["corporate_action"]) == 1 + assert len(sent) == 2 + assert all("corporate_action (1)" in body and "review_approval (1)" in body for body in sent) + assert all("PRIVATE" not in body and "EVENT1" not in body and str(tmp_path) not in body for body in sent) diff --git a/tests/test_dividend_diagnostics.py b/tests/test_dividend_diagnostics.py index 80952e1..6a4e3ca 100644 --- a/tests/test_dividend_diagnostics.py +++ b/tests/test_dividend_diagnostics.py @@ -1288,3 +1288,30 @@ def test_coherent_dividend_keep_quantity_has_no_tolerance(monkeypatch, source_qu if label != "Keep": assert "Action: Compare activity dividend-1" in block assert "compare the quantity by hand" in block + + +@pytest.mark.parametrize("price,expected", [(0.25, False), (0.26, True)]) +def test_operator_action_diagnostic_warning_and_summary(monkeypatch, caplog, price, expected): + from ghostfolio_ibkr_sync.outcome import new_run_outcome + out = new_run_outcome() + findings = comparison(monkeypatch, [destination(unitPrice=price)]) + counts = diagnostics.log_historical_dividend_comparison(findings, ACC, outcome=out) + assert bool(out["actions"].get("dividend_check")) == expected + blocks = [r for r in caplog.records if "Dividend to check:" in r.message] + assert bool(blocks) == expected + assert all(r.levelno >= logging.WARNING for r in blocks) + if expected: + assert counts["to_check"] > 0 + assert all(r.message.count("Action:") == 1 for r in blocks) + + +def test_operator_action_diagnostic_limitation_has_no_impossible_reminder(monkeypatch, caplog): + caplog.set_level(logging.INFO) + from ghostfolio_ibkr_sync.outcome import new_run_outcome + out = new_run_outcome() + xml = report(divs=div_xml(amount="15") + div_xml(amount="10").replace('actionID="A1"', 'actionID="A2"')) + findings = comparison(monkeypatch, [destination(unitPrice=0.15), destination(id="dividend-2", unitPrice=0.10)], xml) + assert findings[0]["reason"] == "component partition identity not proven" + diagnostics.log_historical_dividend_comparison(findings, ACC, outcome=out) + assert out["actions"] == {} + assert "Evidence limitation" in caplog.text diff --git a/tests/test_entrypoint.py b/tests/test_entrypoint.py index 35f8bdb..1f2f186 100644 --- a/tests/test_entrypoint.py +++ b/tests/test_entrypoint.py @@ -34,6 +34,8 @@ def run_entrypoint(tmp_path): ' fi\n' 'else\n' ' echo run >> "$CALLS"\n' + ' printf "%s\\n" "$*" >> "$RUNTIME_DIR/scheduler-args"\n' + ' shift 2\n' ' umask > "$RUNTIME_DIR/runtime-umask"\n' ' cat "$1"\n' 'fi\n' @@ -88,7 +90,7 @@ def test_invalid_cron_is_rejected_without_echoing_input(run_entrypoint, cron): result, crontab, calls = run_entrypoint(cron) assert result.returncode == 1 assert result.stderr == 'Invalid CRON: expected five numeric schedule fields\n' - assert result.stdout == 'ghostfolio-ibkr-sync version test\n' + assert result.stdout == 'ghostfolio-ibkr-sync build test\n' assert not crontab.exists() assert not calls.exists() @@ -97,7 +99,7 @@ def test_supercronic_validation_failure_prevents_start(run_entrypoint): result, crontab, calls = run_entrypoint('99 99 * * *', validation_status=1) assert result.returncode == 1 assert result.stderr == 'Invalid CRON: schedule validation failed\n' - assert result.stdout == 'ghostfolio-ibkr-sync version test\n' + assert result.stdout == 'ghostfolio-ibkr-sync build test\n' assert calls.read_text().splitlines() == ['test'] @@ -136,3 +138,10 @@ def test_no_cron_preserves_one_off_run(run_entrypoint, cron): assert result.returncode == 0 assert calls.read_text() == 'python -m ghostfolio_ibkr_sync\n' assert not crontab.exists() + + +def test_operator_action_logging_scheduler_flags_and_build(run_entrypoint): + result, crontab, calls = run_entrypoint('* * * * *') + assert result.returncode == 0 + assert 'ghostfolio-ibkr-sync build test' in result.stdout + assert '-passthrough-logs -split-logs' in (calls.parent / 'scheduler-args').read_text() diff --git a/tests/test_filters.py b/tests/test_filters.py index aa56054..c4aa5a9 100644 --- a/tests/test_filters.py +++ b/tests/test_filters.py @@ -374,3 +374,16 @@ def test_v2_accepted_state_split_manual_fills_remain_source_scoped(side): assert gate(trades, pos, source_account="U1") == [] assert ids(gate(trades, pos, source_account="U2")) == ["T1", "T2"] assert pos["qty"][(ACC, "KO")] == 100 + + +@pytest.mark.parametrize("kind,reason", [("manual", "trade_manual"), ("symbol", "mapping_conflict"), + ("negative", "trade_position")]) +def test_operator_action_inventory_trade_decisions(kind, reason): + from ghostfolio_ibkr_sync.outcome import new_run_outcome + out = new_run_outcome() + pos = (positions(manual_buys={(ACC, "KO"): [("2026-08-01", 5)]}) if kind == "manual" else + positions(qty={(ACC, "OTHER"): 10}, isin_symbols={(ACC, ISIN): {"OTHER"}}) if kind == "symbol" else + positions(qty={(ACC, "KO"): 1})) + trade = tr("T1", 5 if kind == "manual" else -5) + kept = sync.filter_trades_by_holdings([trade], pos, ACC, {}, set(), source_account="U1", outcome=out) + assert kept == [] and len(out["actions"][reason]) == 1 diff --git a/tests/test_finalize.py b/tests/test_finalize.py index d2268b5..b051f2d 100644 --- a/tests/test_finalize.py +++ b/tests/test_finalize.py @@ -263,7 +263,7 @@ def test_real_hanging_worker_cannot_change_the_outcome(env, tmp_path, caplog): @pytest.mark.parametrize("mode", ["missing", "empty", "conflict", "short"]) -def test_dividend_structured_input_rejections_keep_trades_cash_exit_and_notifications(env, mode): +def test_dividend_structured_input_rejections_keep_trades_cash_exit_and_notify(env, mode): xml = structured_input_xml(mode) w = World(env, xml) @@ -271,7 +271,7 @@ def test_dividend_structured_input_rejections_keep_trades_cash_exit_and_notifica env.setattr(app, "ghost_get_existing_orders", lambda cfg: (set(), set(), positions())) sent = spy_send(env) assert app.main() == 0 - assert sent == [] + assert len(sent) == 1 and "dividend_rejected" in sent[0]["body"] assert len(w.cash_calls) == 3 assert any(a["type"] == "BUY" for a in w.activities) dividends = [a for a in w.activities if a["type"] == "DIVIDEND"] @@ -307,7 +307,7 @@ def test_historical_diagnostic_findings_preserve_success_and_notifications(env, for verb in ("post", "put", "patch", "delete"): env.setattr(requests, verb, lambda *a, **kw: pytest.fail("financial HTTP write")) caplog.set_level(logging.INFO) - assert app.main() == 0 and sent == [] + assert app.main() == 0 and len(sent) == 1 assert "Historical dividends" in caplog.text @@ -366,15 +366,15 @@ def test_historical_diagnostic_preflight_valid_tz_keeps_live_run_silent(env, cap assert "forced to DRY_RUN" not in caplog.text -def test_historical_diagnostic_preflight_raw_dry_run_stays_silent(env): +def test_historical_diagnostic_preflight_raw_dry_run_reports_timezone(env): env.delenv("TZ") env.setenv("DRY_RUN", "1") sent = spy_send(env) accounts(env, ALL_OK) - assert app.main() == 0 and sent == [] + assert app.main() == 0 and len(sent) == 1 -def test_historical_diagnostic_actionable_blocks_never_reach_notifications(env, caplog): +def test_historical_diagnostic_actionable_blocks_join_notifications(env, caplog): env.setenv("DRY_RUN", "1") ids, comments, pos = read(env, [destination(comment="legacy", date="2026-07-17T00:00:00Z")]) xml = report(divs=div_xml()) @@ -383,7 +383,7 @@ def test_historical_diagnostic_actionable_blocks_never_reach_notifications(env, env.setattr(app, "ghost_get_existing_orders", lambda cfg: (ids, comments, pos)) sent = spy_send(env) caplog.set_level(logging.INFO) - assert app.main() == 0 and sent == [] + assert app.main() == 0 and len(sent) == 1 assert "skipped by proximity" in caplog.text and "failed_checks=" in caplog.text @@ -412,7 +412,7 @@ def test_dry_run_modes_notification_compatibility(env, mode, failed, configured, invalid = mode == "invalid" assert app.main() == int(failed or invalid) assert calls == ([] if invalid else ["U111", "U222", "U333"]) - should_notify = configured and (failed or invalid or mode == "false") + should_notify = configured assert len(sent) == int(should_notify) if sent: title = notify.NOTIFY_TITLE if failed or invalid else ( @@ -436,3 +436,125 @@ def test_dry_run_modes_delivery_failure_keeps_exit(env, caplog, mode, status): assert app.main() == 1 and len(sent) == 1 assert calls == ([] if mode == "invalid" else ["U111", "U222", "U333"]) assert URL_SECRET not in caplog.text + + +@pytest.mark.parametrize("mode", [None, "", "true", "false"]) +def test_operator_action_reminders_repeat_then_resolve(env, mode, caplog): + if mode is None: + env.delenv("DRY_RUN") + else: + env.setenv("DRY_RUN", mode) + sent = spy_send(env) + caplog.set_level(logging.INFO) + for _ in range(2): + out = outcome.new_run_outcome() + outcome.record_action(out, "review_approval", "private-event") + outcome.record_action(out, "review_approval", "private-event") + app.finalize_run(out, 0) + assert len(sent) == 2 + assert "review_approval (1)" in sent[0]["body"] + assert "private-event" not in sent[0]["body"] + app.finalize_run(outcome.new_run_outcome(), 0) + assert len(sent) == 2 + assert "Notification result: not required" in caplog.text + + +def test_operator_action_summary_redacts_and_keeps_failure_priority(env): + sent = spy_send(env) + out = outcome.new_run_outcome() + for reason in ("review_approval", "mapping_unverified", "dividend_check"): + outcome.record_action(out, reason, CANARY + AMOUNT) + outcome.record_failure(out, "account", "account_failed", 1) + app.finalize_run(out, 1) + assert len(sent) == 1 and sent[0]["title"] == notify.NOTIFY_TITLE + assert all(reason in sent[0]["body"] for reason in out["actions"]) + assert CANARY not in sent[0]["body"] and AMOUNT not in sent[0]["body"] + + +@pytest.mark.parametrize("raw,status,expected", [ + (None, "sent", "disabled"), ("[]", "sent", "disabled"), + ("invalid-SECRET", "sent", "invalid configuration"), + (json.dumps(URLS), "sent", "sent"), + (json.dumps(URLS), "failed", "failed"), + (json.dumps(URLS), "timeout", "timeout, delivery unconfirmed"), + (json.dumps(URLS), "invalid_destination", "invalid configuration"), + (json.dumps(URLS), RuntimeError(URL_SECRET), "failed"), +]) +def test_operator_action_delivery_explicit(env, caplog, raw, status, expected): + caplog.set_level(logging.INFO) + if raw is None: + env.delenv("APPRISE_URLS") + else: + env.setenv("APPRISE_URLS", raw) + sent = spy_send(env, status) + out = outcome.new_run_outcome() + outcome.record_action(out, "review_approval", "private") + app.finalize_run(out, 0) + assert f"Notification result: {expected}" in caplog.text + assert len(sent) == int(raw == json.dumps(URLS)) + assert URL_SECRET not in caplog.text and "invalid-SECRET" not in caplog.text + assert out["delivery"] == expected + assert "Token expiry notification" not in caplog.text + if raw == json.dumps(URLS): + assert "User action notification" in caplog.text + + +@pytest.mark.parametrize("exc", [KeyboardInterrupt, SystemExit]) +def test_operator_action_delivery_interrupted(env, caplog, exc): + sent = spy_send(env, exc()) + out = outcome.new_run_outcome() + outcome.record_action(out, "review_approval", "private") + with pytest.raises(exc): + app.finalize_run(out, 0) + assert len(sent) == 1 and "Notification result:" not in caplog.text + + +@pytest.mark.parametrize("mode", [None, "", "true", "false"]) +def test_operator_action_prerequisite_timezone_in_simulation(env, mode): + if mode is None: + env.delenv("DRY_RUN") + else: + env.setenv("DRY_RUN", mode) + env.delenv("TZ") + sent = spy_send(env) + accounts(env, ALL_OK) + assert app.main() == 0 and len(sent) == 1 + assert "timezone_correction (1)" in sent[0]["body"] + + + +@pytest.mark.parametrize("failed", [False, True]) +def test_operator_action_integrated_cycles_keep_financial_requests_and_resolve(env, caplog, failed): + from _sync_helpers import CFG, ACC, ISIN_KO, trade_xml + from ghostfolio_ibkr_sync import corporate_actions as ca + caplog.set_level(logging.INFO) + sent = spy_send(env, "failed") + xml = report(trade_xml("T1"), div_xml()) + world = World(env, xml) + for resolved in (False, False, True): + ids, comments, pos = read(env, [destination(unitPrice=0.25 if resolved else 0.26)]) + out = outcome.new_run_outcome() + out["review_held"] = set() if resolved else {"IBKR#v2#U1:TRADE:T1"} + out["deferred"] = set() + if not resolved: + outcome.record_action(out, "review_approval", "IBKR#v2#U1:TRADE:T1") + ca.prepare_notifications(out, None, {"U1": { + "account_held": False, "symbols": set(), "effective": {}, "dates": {}, "normalized": set(), + "findings": {"private-event": {"held": False, "reasons": [], + "conclusions": {"history": "bounded_cost_dates_differ"}, "symbols": ["KO"]}}}}) + if failed: + outcome.record_failure(out, "account", "account_failed", 1) + cfg = {**CFG, "dry_run": True, "_corporate_outcome": out, "_review_holds": out["review_held"]} + _, ok = app.process_account(cfg, "U1", "q1", "IBKR", {ISIN_KO: "KO"} if resolved else {}, + {"T1"}, comments, pos) + assert ok + app.finalize_run(out, int(failed and not resolved)) + assert world.activities == [] + if not resolved: + assert {"review_approval", "mapping_unverified", "dividend_check", "corporate_action"} <= set(out["actions"]) + assert out["delivery"] == "failed" + else: + assert out["actions"] == {} and out["delivery"] == "not required" + assert len(sent) == 2 and len(world.cash_calls) == 3 + assert all("U1" not in item["body"] and ISIN_KO not in item["body"] for item in sent) + assert ("Cycle result: failure" if failed else "success with user action required") in caplog.text diff --git a/tests/test_logging.py b/tests/test_logging.py new file mode 100644 index 0000000..678a5d3 --- /dev/null +++ b/tests/test_logging.py @@ -0,0 +1,41 @@ +"""Application log streams through the CLI setup, with synthetic work.""" +import json +import os +import subprocess +import sys + +import pytest + + +@pytest.mark.parametrize("level", ["DEBUG", "INFO", "WARNING"]) +def test_operator_action_logging_streams_and_evidence(level): + code = """ +import logging +from ghostfolio_ibkr_sync import __main__ as cli +from ghostfolio_ibkr_sync.ibkr import log_lab_evidence + +def synthetic(): + for level in (logging.DEBUG, logging.INFO, logging.WARNING, logging.ERROR, logging.CRITICAL): + logging.getLogger('synthetic').log(level, 'CANARY_LEVEL_%s', logging.getLevelName(level)) + logging.getLogger('urllib3').debug('BROKER_SECRET') + log_lab_evidence('window', {'line': 'one\\ntwo'}) + return 0 +cli.main = synthetic +raise SystemExit(cli.run()) +""" + env = {"PATH": os.environ["PATH"], "LOG_LEVEL": level} + result = subprocess.run([sys.executable, "-c", code], env=env, capture_output=True, text=True, timeout=10) + assert result.returncode == 0, result.stderr + for name in ("DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"): + marker = "CANARY_LEVEL_" + name + visible = (level == "DEBUG" or name != "DEBUG") and (level != "WARNING" or name not in ("DEBUG", "INFO")) + stream = result.stdout if name in ("DEBUG", "INFO") else result.stderr + other = result.stderr if name in ("DEBUG", "INFO") else result.stdout + assert stream.count(marker) == int(visible) + assert marker not in other + assert "BROKER_SECRET" not in result.stdout + result.stderr + lines = [line for line in result.stdout.splitlines() if "LAB_EVIDENCE " in line] + assert len(lines) == int(level == "DEBUG") + if lines: + assert "[DEBUG] LAB_EVIDENCE " in lines[0] + assert json.loads(lines[0].split("LAB_EVIDENCE ", 1)[1]) == {"kind": "window", "value": {"line": "one\ntwo"}} diff --git a/tests/test_orchestration.py b/tests/test_orchestration.py index cbfa023..7e87f3b 100644 --- a/tests/test_orchestration.py +++ b/tests/test_orchestration.py @@ -843,7 +843,7 @@ def raise_for_status(self): raise requests.HTTPError(f"{self.status_code} for url: https://x/?t=SECRET-TOK") -def test_ibkr_get_retries_network_errors_then_succeeds(monkeypatch): +def test_operator_action_inventory_retry_recovers_without_user_action(monkeypatch): attempts, sleeps = [], [] def get(url, params, timeout, **kwargs): attempts.append(1) @@ -1400,7 +1400,8 @@ def test_historical_diagnostic_coherent_import_warnings_only_for_candidates(monk caplog.set_level(logging.INFO) monkeypatch.setattr(app, "ghost_update_cash_balance", lambda *a: True) # cash guard has its own tests assert app.process_account({**CFG, "dry_run": True}, "U1", "q1", "IBKR", {ISIN_KO: "KO"}, ids, comments, pos) == ({}, True) - warnings = [record for record in caplog.records if record.levelno >= logging.WARNING] + warnings = [record for record in caplog.records if record.levelno >= logging.WARNING + and record.name != "ghostfolio_ibkr_sync.diagnostics"] assert len(warnings) == (0 if existing else 1) assert "Historical dividends" in caplog.text if existing: @@ -2002,3 +2003,73 @@ def test_v2_source_qualification_retained_identity_is_not_financial_reconciliati assert world.run(pos=world.pos)[1] is True assert world.activities == [] # Exact stored identities remain trusted. # This limit cannot certify that an unobserved vendor revision is safe. + + +@pytest.mark.parametrize("existing", [set(), {"T1"}]) +def test_operator_action_inventory_mapping_trade_survives_dedup(existing): + from ghostfolio_ibkr_sync.outcome import new_run_outcome + out = new_run_outcome() + for source in ("U1", "U2"): + xml = report(trade_xml("T1")).replace('accountId="U1"', f'accountId="{source}"') + for _ in range(2): + activities, _, _, ok = app.select_account_activities( + xml, source, ACC, {}, set(existing), set(), positions(), outcome=out) + assert ok + assert len(activities) == int(not existing) + assert len(out["actions"]["mapping_unverified"]) == 2 + resolved = new_run_outcome() + app.select_account_activities(report(trade_xml("T1")), "U1", ACC, {ISIN_KO: "KO"}, + {"T1"}, set(), positions(), outcome=resolved) + assert resolved["actions"] == {} + + +def test_operator_action_inventory_trade_invalid_and_normal_exclusions(): + from ghostfolio_ibkr_sync.outcome import new_run_outcome + out = new_run_outcome() + activities, _, _, ok = app.select_account_activities( + report(trade_xml("T1").replace('tradePrice="60"', 'tradePrice="bad"')), + "U1", ACC, {ISIN_KO: "KO"}, set(), set(), positions(), outcome=out) + assert not ok and not activities + assert out["actions"]["trade_invalid"] == {"U1:T1"} + + +@pytest.mark.parametrize("mode", ["missing", "empty", "conflict", "short"]) +def test_operator_action_inventory_dividend_selection_unchanged(mode): + from ghostfolio_ibkr_sync.outcome import new_run_outcome + xml = structured_input_xml(mode) + out = new_run_outcome() + args = (xml, "U1", ACC, {ISIN_KO: "KO", ISIN_PEP: "PEP"}, set(), set()) + baseline = app.select_account_activities(*args, positions()) + reported = app.select_account_activities(*args, positions(), outcome=out) + assert reported == baseline + assert out["actions"]["dividend_rejected"] + + +@pytest.mark.parametrize("dry,cash_ok,missing,expected", [(False, True, False, "request reported successful"), + (True, True, False, "simulated"), (False, False, False, "uncertain/failed"), + (False, True, True, "not updated")]) +def test_operator_action_result_independent_cash(monkeypatch, caplog, dry, cash_ok, missing, expected): + from ghostfolio_ibkr_sync.outcome import new_run_outcome + caplog.set_level(logging.INFO) + xml = report(trade_xml("T1")) + if missing: + xml = xml.replace('currency="BASE_SUMMARY"', 'currency="USD"') + w = World(monkeypatch, xml, cash_ok=cash_ok) + out = new_run_outcome() + cfg = {**CFG, "dry_run": dry, "_corporate_outcome": out} + _, ok = app.process_account(cfg, "U1", "q1", "IBKR", {ISIN_KO: "KO"}, set(), set(), positions()) + app.finalize_run(out, int(not ok)) + assert expected in caplog.text + assert ("simulated proposals=1" if dry else "accepted imports=1") in caplog.text + assert "independent balance readback" not in caplog.text + assert len(w.cash_calls) == int(not missing) + + +def test_operator_action_result_early_failure_has_no_invented_zeros(monkeypatch, caplog): + from ghostfolio_ibkr_sync.outcome import new_run_outcome + caplog.set_level(logging.INFO) + out = new_run_outcome() + app.finalize_run(out, 1) + assert "Cycle result: failure" in caplog.text + assert "imports=unavailable" in caplog.text and "cash=unavailable" in caplog.text + assert "notification=disabled" in caplog.text diff --git a/tests/test_outcome.py b/tests/test_outcome.py index 2f5957d..2b1918f 100644 --- a/tests/test_outcome.py +++ b/tests/test_outcome.py @@ -100,7 +100,8 @@ def test_clean_run_records_no_failure(env): holder = capture_outcome(env) stub_accounts(env, {"U111": ({}, True), "U222": ({}, True), "U333": ({}, True)}) assert app.main() == 0 - assert holder["outcome"] == {"failures": [], "dropped": 0, "accounts_total": 3, "accounts_failed": 0} + assert {key: holder["outcome"][key] for key in ("failures", "dropped", "accounts_total", "accounts_failed")} == {"failures": [], "dropped": 0, "accounts_total": 3, "accounts_failed": 0} + assert holder["outcome"]["actions"] == {} def test_a_failed_run_leaves_nothing_in_the_next_run_outcome(env): @@ -112,7 +113,8 @@ def test_a_failed_run_leaves_nothing_in_the_next_run_outcome(env): stub_accounts(env, {"U111": ({}, True), "U222": ({}, True), "U333": ({}, True)}) assert app.main() == 0 assert holder["outcome"] is not first - assert holder["outcome"] == {"failures": [], "dropped": 0, "accounts_total": 3, "accounts_failed": 0} + assert {key: holder["outcome"][key] for key in ("failures", "dropped", "accounts_total", "accounts_failed")} == {"failures": [], "dropped": 0, "accounts_total": 3, "accounts_failed": 0} + assert holder["outcome"]["actions"] == {} def test_warning_only_run_records_no_failure(env): @@ -204,7 +206,8 @@ def process(cfg, ibkr_id, *a): {"stage": "account", "reason": "unexpected_error", "ordinal": 1}, {"stage": "account", "reason": "account_failed", "ordinal": 3}] assert (out["accounts_total"], out["accounts_failed"]) == (3, 2) - text = json.dumps(out) + from ghostfolio_ibkr_sync.notify import build_failure_summary + text = build_failure_summary(out) for leak in (CANARY, AMOUNT, "U111", "U333", "ghost:3333", "KeyError"): assert leak not in text diff --git a/tests/test_token_expiry.py b/tests/test_token_expiry.py index ecf0301..850bf3e 100644 --- a/tests/test_token_expiry.py +++ b/tests/test_token_expiry.py @@ -151,13 +151,14 @@ def test_reminder_delivery_problem_does_not_fail_sync_or_leak(env, caplog, statu @pytest.mark.parametrize("mode", [None, "", " ", "1", "true", "yes", "on"]) -def test_dry_run_modes_expiry_only_warns_without_notification(env, caplog, mode): +def test_operator_action_prerequisite_dry_run_modes_expiry_notifies(env, caplog, mode): if mode is None: env.delenv("DRY_RUN") else: env.setenv("DRY_RUN", mode) sent = notifications(env) - assert app.main() == 0 and sent == [] + assert app.main() == 0 and len(sent) == 1 + assert "token_renewal (1)" in sent[0][1] assert "expires in 3 day(s)" in caplog.text