DTSB — Discretionary Trading Strategy Backtester

Backtest AI / LLM discretionary trading strategies — strategies that are judgment (a prompt) rather than fixed rules — with no-lookahead point-in-time walking, anti-contamination controls, and deterministic outcome scoring. Bring your own LLM: a free OpenRouter model (auto health-checked) or any OpenAI-compatible endpoint (OpenAI, Ollama, LM Studio, vLLM, a paid model — your choice).

Not an MCP server, not a live-trading bot. It generates and scores historical signals only.

Why this needs to be more than "loop an LLM over some bars"

A discretionary strategy — "buy pullbacks in an uptrend," "fade RSI extremes in a range" — has no equation to hand to a Strategy Tester or a vectorized Python loop. The only way to backtest it is to actually ask the model, bar by bar, what it would have done, then score the resulting trades. That's easy to get wrong in ways a mechanical backtest never fails on:

Failure mode	What goes wrong	DTSB's control
Training-data contamination	The model may have seen the test period during training and "predicts" the future by remembering it	Packets are anonymized (symbol → alias, prices scaled, dates shifted) before the model ever sees them, plus an explicit contamination probe
Lookahead	Using bar data that wouldn't have existed yet at decision time	A point-in-time walker builds each packet from only bars closed by that instant; unit-tested
Self-grading	The LLM decides both the trade AND whether it won	The LLM only generates ballots; a deterministic walker scores outcomes against real bars
Non-determinism	The same prompt can return a different verdict on repeat	A variance probe measures the flip rate and reports it as a noise band
Overfitting / small samples	Months of LLM-scale data ≠ statistical power	Chronological train/test split; bootstrap confidence intervals on the test split only

Full writeup: docs/LLM-VS-MECHANICAL.md. Architecture: docs/DESIGN.md.

Every run produces report.md — the expectancy/profit-factor numbers alongside this honesty certificate (scout recall, gate miss rate, ballot validity, and once you run the probes, vote variance and contamination). A result without its certificate isn't a result.

Quickstart A — zero API keys (CSV + local Ollama)

pip install -r requirements.txt
cp .env.example .env                    # nothing to fill in for this path
cp config.example.yaml dtsb.yaml

Put your own OHLCV data in ./data/csv/EURUSD_4h.csv (header: time,open,high,low,close,volume), or point providers.csv.dir at wherever you keep it. Then in dtsb.yaml:

provider: csv
providers: { csv: { dir: ./data/csv } }
engine:
  kind: openai_compat
  openai_compat: { base_url: "http://localhost:11434/v1", model: "llama3.3" }

python -m dtsb.backtest.driver --config dtsb.yaml --check-deps   # sanity check
python -m dtsb.backtest.driver --config dtsb.yaml --limit 20     # small first run
python -m dtsb.backtest.bridge  <run_id>     # optional: export a signal CSV
python -m dtsb.backtest.score   <run_id>     # expectancy / PF / bootstrap CIs
python -m dtsb.backtest.report  <run_id>     # writes data/runs/<run_id>/report.md

Quickstart B — TradingView data + a free LLM

provider: tradingview                # needs TVR_API_KEY in .env (https://tvremix.xyz)
engine:
  kind: openrouter                   # needs OPENROUTER_API_KEY in .env
  openrouter: { models: [qwen/qwen3-next-80b-a3b-instruct:free] }

The openrouter engine probes candidates with a real ballot-shaped prompt (not a trivial ping — some models answer a hello fine but fail to follow a "respond with only JSON" instruction), remembers which ones are healthy, and learns your actual daily free-request cap from OpenRouter's own 429 response rather than assuming a number.

Writing a strategy

A strategy is a directory — see strategies/solo-swing/ (one persona, always convenes) and strategies/mini-council/ (three personas + weighted consensus, each gated by its own scout triggers) for working examples.

strategies/my-strategy/
  strategy.yaml
  my_persona.md

strategy.yaml:

name: my-strategy
personas: [my_persona]
aggregation: { consensus_threshold: 0.6, min_conviction: 0.0 }

my_persona.md:

---
id: my_persona
display_name: "My Persona"
needs: [rsi14, adx14]          # indicator ids -> dtsb/core/indicators.py
triggers:                       # optional -- omit to always convene
  - {kind: adx_min, value: 25}
---
Your method, written as you'd explain it to a trader. This becomes the
prompt body the LLM reasons from.

Available indicator ids and scout trigger kinds are listed in dtsb/core/indicators.py and dtsb/backtest/scout.py's module docstrings respectively.

Honest warnings

• Results are pinned to one model version — a model upgrade invalidates the estimate until you re-run.
• Confidence intervals from a few months of history are wide. If the CI spans zero R, the report says so plainly rather than overclaiming an edge.
• Anonymization reduces contamination risk; it does not prove zero.
• A backtest — mechanical or LLM — is still a backtest: regime change and execution reality aren't in the numbers.

Project layout

dtsb/
  core/        packet building, indicators, ballot schema, aggregation,
               SL/TP outcome resolution, prompt templates
  providers/   csv | tradingview (tvremix) | mt5 data sources
  engines/     openai_compat (generic) | openrouter (free-model pool)
  strategy.py  the strategy adapter (load a strategies/<name>/ directory)
  backtest/    walk, scout, reduce, anonymize, gate, validate, driver,
               score, bridge, report, probes
strategies/    example strategies (solo-swing, mini-council)
tests/         pytest -- no network required, all HTTP mocked
docs/          design + the LLM-vs-mechanical rationale

Non-goals (v1)

No live order execution — signals-only. No multi-symbol portfolio effects. No parameter-optimization loops beyond the single consensus-threshold / min-conviction knobs a strategy author sets themselves.

License

MIT — see LICENSE.