feat: ground trader in market data and harden unattended runs

.github/workflows/ci.yml

@@ -0,0 +1,29 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+
+      - name: Sync workspace
+        run: uv sync --all-packages
+
+      - name: Ruff lint
+        run: uv run ruff check .
+
+      - name: Ruff format check
+        run: uv run ruff format --check .
+
+      - name: Pytest (coverage threshold enforced by pytest config)
+        run: uv run pytest

.github/workflows/docker.yml

@@ -0,0 +1,38 @@
+# Builds the pipeline/MCP image on main and pushes it to GHCR, so the CasaOS
+# host can `docker compose pull` instead of building ~1GB of deps locally.
+name: Docker image
+
+on:
+  push:
+    branches: [main]
+  workflow_dispatch:
+
+jobs:
+  build-push:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      packages: write
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Log in to GHCR
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Set up Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Build and push
+        uses: docker/build-push-action@v6
+        with:
+          context: .
+          push: true
+          tags: |
+            ghcr.io/${{ github.repository }}:latest
+            ghcr.io/${{ github.repository }}:${{ github.sha }}
+          cache-from: type=gha
+          cache-to: type=gha,mode=max

README.md

@@ -59,24 +59,27 @@ Fetches cryptocurrency quotes and OHLC history from CoinGecko, then generates AS
 
 ### `/financial-news`
 
-Fetches the latest financial news from 10 RSS feeds (MarketWatch, Yahoo Finance, CNBC, Investing.com, Motley Fool) and ranks articles by keyword match and semantic similarity using a local sentence-transformer model.
+Fetches the latest financial news from 10 RSS feeds (MarketWatch, Yahoo Finance, CNBC, Investing.com, Motley Fool), removes duplicates syndicated across feeds, and ranks articles by keyword match and semantic similarity using a local sentence-transformer model.
 
 **Usage**
 
 ```
 /financial-news
 /financial-news --hours 6
+/financial-news --min-semantic 0.40
 ```
 
-| Argument    | Required | Default | Description                              |
-|-------------|----------|---------|------------------------------------------|
-| `--hours`   | No       | `24`    | Look-back window in hours                |
-| `--keywords`| No       | —       | Path to a `keywords.json` config file    |
+| Argument         | Required | Default | Description                                    |
+|------------------|----------|---------|------------------------------------------------|
+| `--hours`        | No       | `24`    | Look-back window in hours                      |
+| `--keywords`     | No       | —       | Path to a `keywords.json` config file          |
+| `--min-semantic` | No       | —       | Only include articles with semantic score ≥ N  |
+| `--min-keyword`  | No       | —       | Only include articles with keyword score ≥ N   |
 
 **What it returns**
 
-- Fetched articles grouped by source, newest first
-- Ranked list with keyword score (0–1) and semantic score (0–1)
+- Fetched articles grouped by source, newest first, deduplicated by URL/title
+- Ranked list with keyword score (0–1; capped count — 1 match = 0.33, 3+ = 1.0) and semantic score (0–1)
 - Group 1 — keyword match **and** semantic score ≥ 0.45
 - Group 2 — semantic score ≥ 0.40 with no keyword match
 - An observation paragraph summarising score spread and relevance
@@ -123,18 +126,40 @@ No arguments. Requires `CRYPTO_API_KEY` and `CRYPTO_API_SECRET` environment vari
 
 ---
 
-### `/financial-news` — scoring filter
+### `/trader`
+
+Runs a multi-agent investment analysis for a symbol: a five-role analyst panel (fundamental, technical, macro, sentiment, risk) in parallel, a fixed-rounds bull/bear debate over the panel's findings, and an impartial judge that scores the debate and returns a structured verdict.
 
-The `analyze_news.py` script now accepts `--min-semantic` and `--min-keyword` flags to pre-filter articles before writing output, eliminating ad-hoc filtering steps.
+**Usage**
 
 ```
-/financial-news --min-semantic 0.40
+/trader --symbol AMZN
 ```
 
-| Argument         | Required | Default | Description                                    |
-|------------------|----------|---------|------------------------------------------------|
-| `--min-semantic` | No       | —       | Only include articles with semantic score ≥ N  |
-| `--min-keyword`  | No       | —       | Only include articles with keyword score ≥ N   |
+| Argument         | Required | Default | Description                                        |
+|------------------|----------|---------|----------------------------------------------------|
+| `--symbol`       | Yes      |         | Ticker symbol to analyse                           |
+| `--context-file` | No       | —       | Market data file to ground the panel (quote/history) |
+| `--rounds`       | No       | `2`     | Debate rounds                                      |
+
+**What it returns**
+
+- Panel briefs, debate transcript, and a judge scorecard (evidence quality, rebuttal validity, risk-adjusted logic, internal consistency, falsifiability) with winner, confidence, and an actionable verdict
+- The verdict JSON is schema-enforced via structured outputs
+
+> Each full run makes 10 Anthropic API calls (5× Sonnet panel, 4× Opus debate, 1× Opus judge), measured at ~$0.16. Models and prompts live in `skills/trader/config/roles.yaml`.
+
+---
+
+### `/daily-pipeline`
+
+Orchestrates everything into one checkpointed daily run: fetch + rank news, extract top ticker mentions, fetch portfolios, ground the trader with `check_stock` market data, run it on the most relevant symbols, build a Markdown report (with token usage and estimated cost), and email it. Re-running the same day resumes from the first incomplete stage; on failure it sends a best-effort alert email.
+
+```
+uv run skills/daily_pipeline/scripts/run_daily.py [--dry-run] [--skip-email] [--no-summary]
+```
+
+Thresholds, caps, and pricing live in `skills/daily_pipeline/pipeline.yaml`. See `skills/daily_pipeline/SKILL.md` for the full stage list and `docs/deployment.md` for running it on a schedule.
 
 ---
 
@@ -166,7 +191,7 @@ uv run python --version
 test -d .venv && echo "venv OK"
 
 # Key packages
-uv run python -c "import check_stock, check_crypto, financial_news, get_stock_portfolio, get_crypto_portfolio, common" && echo "All packages OK"
+uv run python -c "import check_stock, check_crypto, financial_news, get_stock_portfolio, get_crypto_portfolio, trader, daily_pipeline, mcp_server, common" && echo "All packages OK"
 ```
 
 ---

common/src/common/http_client.py

@@ -1,9 +1,19 @@
 """Generic HTTP client using requests.Session, shared across all skills."""
 
 import requests
+from requests.adapters import HTTPAdapter
+from urllib3.util.retry import Retry
 
 from common.logger import get_logger
 
+USER_AGENT = "finance-agent/0.1 (+https://github.com/agb1986/finance-agent)"
+
+# The pipeline runs unattended, so transient failures (CoinGecko 429s, feed
+# hiccups, 5xx blips) must not fail a stage outright. Retry respects Retry-After.
+DEFAULT_RETRIES = 3
+DEFAULT_BACKOFF = 1.0
+RETRY_STATUSES = (429, 500, 502, 503, 504)
+
 
 class HttpClient:
     """Thin wrapper around requests.Session with a fixed base URL and shared headers."""
@@ -13,12 +23,24 @@ def __init__(
         base_url: str,
         headers: dict[str, str] | None = None,
         timeout: int = 30,
+        retries: int = DEFAULT_RETRIES,
     ) -> None:
         self.base_url = base_url.rstrip("/")
         self.timeout = timeout
         self.session = requests.Session()
+        self.session.headers["User-Agent"] = USER_AGENT
         if headers:
             self.session.headers.update(headers)
+        retry = Retry(
+            total=retries,
+            backoff_factor=DEFAULT_BACKOFF,
+            status_forcelist=RETRY_STATUSES,
+            allowed_methods=("GET", "POST"),
+            respect_retry_after_header=True,
+        )
+        adapter = HTTPAdapter(max_retries=retry)
+        self.session.mount("https://", adapter)
+        self.session.mount("http://", adapter)
 
     def get(self, path: str, params: dict | None = None) -> dict | list:
         """Perform a GET request and return the parsed JSON response.

common/tests/test_http_client.py

@@ -5,7 +5,7 @@
 
 import pytest
 import requests
-from common.http_client import HttpClient
+from common.http_client import RETRY_STATUSES, USER_AGENT, HttpClient
 
 
 @pytest.fixture(autouse=True)
@@ -35,6 +35,27 @@ def test_custom_timeout(self):
         client = HttpClient("https://example.com", timeout=10)
         assert client.timeout == 10
 
+    def test_user_agent_set(self):
+        client = HttpClient("https://example.com")
+        assert client.session.headers["User-Agent"] == USER_AGENT
+
+    def test_custom_headers_do_not_remove_user_agent(self):
+        client = HttpClient("https://example.com", headers={"Authorization": "key123"})
+        assert client.session.headers["User-Agent"] == USER_AGENT
+
+    def test_retry_adapter_mounted(self):
+        client = HttpClient("https://example.com")
+        for prefix in ("https://", "http://"):
+            retry = client.session.get_adapter(f"{prefix}example.com").max_retries
+            assert retry.total == 3
+            assert set(RETRY_STATUSES) <= set(retry.status_forcelist)
+            assert "POST" in retry.allowed_methods
+
+    def test_retry_count_configurable(self):
+        client = HttpClient("https://example.com", retries=5)
+        retry = client.session.get_adapter("https://example.com").max_retries
+        assert retry.total == 5
+
 
 class TestHttpClientGet:
     def _client(self, headers: dict | None = None) -> HttpClient:

docs/deployment.md

@@ -174,9 +174,18 @@ reported on a separate line so they never inflate the day's total.
 | `lock file ... another run in progress?` | A previous run is still going, or died hard. Locks older than 6h are cleared automatically. |
 | Model download attempted at runtime | `HF_HUB_OFFLINE=1` is set because the model is baked in. If you change the embedding model, rebuild the image. |
 
-## Not yet done
+## Failure alerting
 
-- **Failure alerting** — a failed stage currently just exits non-zero into the
-  cron log; there is no push/email notification.
-- **CI/CD** — no GitHub Actions job builds and pushes the image to GHCR yet, so
-  the CasaOS host builds locally.
+When a required stage fails, the pipeline exits non-zero **and** sends a
+best-effort alert email ("Daily Finance Report — pipeline FAILED") using the
+same SMTP settings as the report. If SMTP itself is the failure, the alert is
+skipped and the error stays in the cron log.
+
+## CI/CD
+
+GitHub Actions runs tests + ruff on every PR (`.github/workflows/ci.yml`) and
+builds + pushes the image to GHCR on every merge to main
+(`.github/workflows/docker.yml`). To consume the prebuilt image on the CasaOS
+host instead of building locally, point the compose services at
+`ghcr.io/agb1986/finance-agent:latest` and run `docker compose pull` — the
+local `build:` fallback keeps working either way.

mcp_server/src/mcp_server/server.py

@@ -5,12 +5,17 @@
 import subprocess
 from pathlib import Path
 
+from common.logger import get_logger, setup
 from mcp.server.fastmcp import FastMCP
 
 REPO_ROOT = Path(os.environ.get("REPO_ROOT", Path.cwd()))
 PORT = int(os.environ.get("MCP_PORT", "35001"))
 HOST = os.environ.get("MCP_HOST", "0.0.0.0")
 
+# Mirrors the daily pipeline's per-script cap. Without a timeout, one hung
+# yfinance call would wedge the tool call (and its SSE connection) forever.
+SCRIPT_TIMEOUT = 900
+
 mcp = FastMCP("Finance Agent", host=HOST, port=PORT)
 
 
@@ -21,14 +26,22 @@
 
 def _run(script: str, args: list[str]) -> str:
     """Run a skill script via uv run and return stripped stdout."""
-    result = subprocess.run(
-        ["uv", "run", script, *args],
-        capture_output=True,
-        text=True,
-        cwd=REPO_ROOT,
-    )
+    logger = get_logger()
+    logger.debug(f"running: {script} {' '.join(args)}")
+    try:
+        result = subprocess.run(
+            ["uv", "run", script, *args],
+            capture_output=True,
+            text=True,
+            cwd=REPO_ROOT,
+            timeout=SCRIPT_TIMEOUT,
+        )
+    except subprocess.TimeoutExpired as exc:
+        logger.error(f"{script} timed out after {SCRIPT_TIMEOUT}s")
+        raise RuntimeError(f"Script '{script}' timed out after {SCRIPT_TIMEOUT}s") from exc
     if result.returncode != 0:
         detail = result.stderr.strip() or f"exit code {result.returncode}"
+        logger.error(f"{script} failed: {detail}")
         raise RuntimeError(f"Script '{script}' failed: {detail}")
     return result.stdout.strip()
 
@@ -156,4 +169,5 @@ def analyze_financial_news(hours: int = 24, min_semantic: float = 0.40) -> str:
 # ---------------------------------------------------------------------------
 
 if __name__ == "__main__":
+    setup(debug=os.environ.get("MCP_DEBUG", "") != "")
     mcp.run(transport="sse")

skills/daily_pipeline/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: daily-pipeline
-version: 0.1.2
+version: 0.1.3
 description: Runs the full daily finance pipeline — fetches and ranks news, extracts top ticker mentions, fetches portfolios, runs the trader on the most relevant symbols, builds a Markdown report, and emails it. Use when the user asks for the daily report, the full pipeline, or a morning briefing.
 ---
 
@@ -10,10 +10,16 @@ Orchestrates the other skills into one checkpointed daily run:
 
 ```
 fetch_news → analyze_news → extract_tickers ─┐
-stock_portfolio → select_candidates ─────────┼→ trader (per symbol) → build_report → send_email
-crypto_portfolio (artifact only, v1) ────────┘
+stock_portfolio → select_candidates ─────────┼→ market context → trader (per symbol) → build_report → send_email
+crypto_portfolio (rendered in report) ───────┘
 ```
 
+Before each trader run the pipeline fetches the symbol's quote and history via
+`check_stock` (free Yahoo data) and passes a condensed context file to the
+analyst panel, so verdicts are grounded in current prices rather than training
+data. The lookback is `trader.history_period` in `pipeline.yaml`. If the fetch
+fails the panel still runs, just ungrounded.
+
 Each run writes to `tmp/run_<YYYYMMDD>/` with a `manifest.json` checkpoint —
 re-running the same day resumes from the first incomplete stage, so a crash or
 a cron re-fire never repeats completed API spend. Trader verdicts newer than
@@ -108,8 +114,9 @@ skills/daily_pipeline/tmp/run_20260801/report.md
 ```
 
 If SMTP is not configured, pass `--skip-email`. On stage failure the script
-exits non-zero with the failing stage in the error log — re-running resumes
-from that stage.
+exits non-zero with the failing stage in the error log, and sends a best-effort
+failure alert email using the same SMTP settings as the report — re-running
+resumes from the failed stage.
 
 ### Step 3 — Present the result
 
@@ -127,7 +134,7 @@ Each stage is also runnable standalone (all print their output path on stdout):
 ```bash
 uv run skills/daily_pipeline/scripts/extract_tickers.py --input <analysis.json> [--top-n 5]
 uv run skills/daily_pipeline/scripts/select_candidates.py --input <portfolio.json> [--min-pnl-pct 5]
-uv run skills/daily_pipeline/scripts/build_report.py --analysis <a.json> --tickers <t.json> [--portfolio <p.json>] [--verdict <v.json>]...
+uv run skills/daily_pipeline/scripts/build_report.py --analysis <a.json> --tickers <t.json> [--portfolio <p.json>] [--crypto <c.json>] [--verdict <v.json>]...
 uv run skills/daily_pipeline/scripts/send_email.py --report <report.md> [--subject "..."]
 ```
 
@@ -146,7 +153,5 @@ persist in named volumes rather than the working tree.
 
 ## Known limitations (v1)
 
-- The crypto portfolio is fetched and recorded as an artifact but not yet
-  rendered in the report, and crypto holdings are not gated into trader runs.
-- The trader panel runs without a market-data context file; wiring
-  `check_stock` quotes in as context is a planned improvement.
+- Crypto holdings are rendered in the report but not yet gated into trader
+  runs — only stock positions feed `select_candidates`.

skills/daily_pipeline/pipeline.yaml

@@ -16,6 +16,7 @@ trader:
   max_runs: 2              # hard cap on trader pipelines per day (10 API calls each)
   verdict_cache_days: 3    # reuse an existing verdict newer than this many days
   rounds: 2                # debate rounds passed to run_debate.py
+  history_period: "6mo"    # check_stock lookback used to ground the panel with market data
 
 email:
   subject_prefix: "Daily Finance Report"