Leakage-resistant evaluation for LLM trading agents.
English Β· δΈζ
Run the proof Β· Run the console Β· What's new here Β· Results Β· Limitations Β· ADRs
Most AI trading projects cannot tell you whether their edge is real. The tooling to answer that question β point-in-time discipline, leakage detection, honest attribution of what the LLM actually contributed β is either missing or assumed. This project is that tooling, wrapped around a working A-share / US-equity research and simulated-trading system.
Why "NoAlpha"? Because claiming alpha is easy and proving it is hard. This project refuses to claim what it cannot prove β including about itself.
The fastest way to understand NoAlpha is to watch it manufacture a convincing backtest and then reject it. The first run downloads and caches real market data; subsequent runs are offline and faster.
git clone https://github.com/flysheep-ai/NoAlpha.git
cd NoAlpha
python -m venv .venv && source .venv/bin/activate
pip install -e ".[quant]"
noalpha demo leakageExpected result:
Rank IC Rank ICIR net return sharpe
clean protocol +0.0274 +0.115 +11.45% +0.21
leaky same-bar +0.3644 +1.488 -33.07% -0.22
Rank IC inflation: 13.3x
OK: leakage reproduced and rejected (13.3x inflation).
Same features, folds, costs, and executable returns. Only the label changes. The command exits non-zero if the leak ever stops reproducing.
- π Point-in-time by construction β labels encode their execution assumption (
open[T+1+h]/open[T+1]-1). Same-bar execution is structurally impossible, not merely discouraged. - π°οΈ Forward-only LLM feature collection β an append-only store whose
available_atis the wall-clock end of a real LLM call, and which refuses to backfill: a practical defense against parametric look-ahead. - π§ͺ A backtest that manufactures its own fraud β one command reproduces a 13.3Γ Rank IC inflation from a single swapped label, and exits non-zero if the leak ever stops reproducing.
- π Three-arm attribution β
quant_only/deterministic_hybrid/llm_hybridrun on identical quotes, costs and cycle timing, so "did the LLM help?" is an attributable number rather than a vibe. - π¦ Promotion is a gate, not a habit β
EXPERIMENTAL β VALIDATED β SHADOW β PAPER β ACTIVE, enforced in code. The US model failed to beat its benchmark, so it sits atSHADOWand does not trade. - π§ The LLM never places an order β it emits a structured signal into a versioned deterministic fusion policy that can damp or veto, but not originate, a trade. No
"you are a trader"prompt exists anywhere in the codebase. - π A-shares and US equities β multi-host provider fallback, exchange-timestamp freshness, trading calendars; delay-in-seconds is a first-class API field.
- π₯οΈ A console that admits what it didn't use β FastAPI + React; every position traces back to the decision story that produced it, and unavailable layers are labelled
UNAVAILABLEon screen instead of silently faked. - β»οΈ Reproducible end to end β 240+ tests, including leakage attacks that actively try to sneak future information past the point-in-time guard.
The important thing in this screenshot is not the P&L β it's the two red
UNAVAILABLEbadges. The system is telling you, in its own UI, that the quant model produced no fresh features this cycle and the LLM research layer was never enabled. It has downgraded itself to a deterministic signal strategy and says so, instead of pretending to capabilities it did not use. Most systems bury that line in a log file; here it sits at the top of the screen.
Every position traces back to the decision story and proposal that produced it: which signals fired, at what confidence, under which decision mode. "Quant inference unavailable" is printed on the panel too β no silent fallback.
Screenshots show the Chinese locale; the console ships with an English/Chinese toggle in the sidebar. Some in-page copy and the agent's generated decision narratives are Chinese-only for now.
Four design choices that distinguish this repository from most open-source trading projects:
Guarding the inputs to an LLM is not enough. An LLM asked today what it thought about a stock in 2024 does not reason from the evidence you hand it β it recalls what happened, because the outcome is in its pretraining corpus. The literature calls this parametric look-ahead, and it is an active 2026 research problem:
- Detecting Lookahead Bias in LLM Forecasts β models recall in-cutoff prices verbatim
- Look-Ahead-Bench β significant temporal bias in general-purpose models
- Chronologically Consistent LMs β mitigates it by retraining on time-stamped corpora
The academic fix is to retrain the model, which is out of reach for most
practitioners. noalpha/quant/llm_features.py takes the engineering route
instead: an append-only store whose available_at is the wall-clock completion
time of a real LLM call, which refuses to backfill. Retro-generated rows are
forced to pit_safe=false and excluded from readiness counts.
It is slow β readiness is months, not minutes β because the implementation chooses auditable forward collection over unverifiable historical backfills.
python experiments/leakage_demo.pyTwo identical LightGBM models, identical purged walk-forward folds, identical real A-share bars. One label swapped:
Rank IC Rank ICIR net return sharpe
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
clean protocol +0.0274 +0.115 +11.45% +0.21
leaky same-bar +0.3644 +1.488 -33.07% -0.22
Rank IC inflation: 13.3x
close[T+4]/close[T-1] is one of the most common label formulations in public
backtesting code. It spans decision day T, which the model's own return_1d
feature already knows. It looks 13x better and loses a third of the account.
The demo exits non-zero if the leak stops reproducing, so it runs as a regression test on the point-in-time machinery β not a screenshot.
Three arms run against the same quotes, the same cost assumptions and the same
cycle timing: quant_only, deterministic_hybrid, llm_hybrid. Only one drives
the broker; the others are shadow portfolios. Every divergence is logged with
llm_changed_decision, so the LLM's contribution is an attributable number
rather than a vibe.
Most projects can't answer "did the LLM help?" because they never ran the counterfactual. This one is built to.
EXPERIMENTAL β VALIDATED β SHADOW β PAPER β ACTIVE, one step at a time,
enforced in code. Training a model never activates it. The US model failed to
beat its benchmark, so it sits at SHADOW and does not trade β the gate
working as designed, not an oversight.
Related: the LLM never places an order. It emits a structured signal into a versioned deterministic fusion policy that can damp or veto but not originate a trade. There is no "you are a trader" prompt anywhere in the codebase β the inverse of the prevailing agent-first design.
| Layer | Module | What it does |
|---|---|---|
| Market data | noalpha/market/ |
Multi-host provider fallback, exchange-timestamp freshness, trading calendars |
| Evidence research | noalpha/research/, pit.py |
Point-in-time guard, evidence normalization, timeline, event detection, citation validation |
| Quant | noalpha/quant/ |
Versioned labels, purged walk-forward, experiment store, model registry + promotion gate, calibration, ensembling |
| Trading | noalpha/trading/ |
Autonomous loop, LLM fusion policy, hard risk engine, virtual broker, three-arm shadow comparison |
Labels encode execution assumptions: open[T+1+h]/open[T+1]-1 β decide at T
close, fill at T+1 open. Same-bar execution is structurally impossible rather
than merely discouraged.
Real walk-forward, 2019-01 β 2026-07, 11 folds, expanding train 500d / valid 60d / test 120d / purge 11d, net of cost (5bp commission + 10bp slippage, plus 5bp sell-side stamp duty for A-shares).
| Model | IC | Rank IC | Rank ICIR | Net return | Sharpe | MDD | Turnover |
|---|---|---|---|---|---|---|---|
| random | β0.006 | +0.002 | +0.01 | β47.2% | β0.45 | β54% | 0.83 |
| simple_momentum | +0.021 | +0.018 | +0.06 | +71.5% | +0.58 | β35% | 0.37 |
| simple_reversal | +0.007 | +0.002 | +0.01 | β31.0% | β0.16 | β43% | 0.76 |
prior heuristic (opportunity_score) |
+0.014 | +0.013 | +0.04 | β26.1% | β0.14 | β57% | 0.58 |
| linear_ridge | +0.015 | β0.003 | β0.01 | β31.3% | β0.18 | β47% | 0.49 |
| lightgbm (champion, ACTIVE) | +0.033 | +0.027 | +0.115 | +11.5% | +0.21 | β30% | 0.46 |
| xgboost | +0.030 | +0.035 | +0.154 | β2.4% | +0.09 | β36% | 0.52 |
| equal-weight benchmark | β | β | β | +2.9% | +0.13 | β32% | 0 |
The champion beats the benchmark on return, drawdown and turnover, and loses on return to a five-line momentum rule. That tradeoff is recorded rather than hidden β which is the point of building the measurement layer in the first place.
On US equities no model beat the equal-weight benchmark (+149.8%, Sharpe 1.02),
so none was promoted past SHADOW.
| Strategy | A-shares | US |
|---|---|---|
| Quant only (LightGBM) | +11.5% | +145.7% |
| Deterministic research proxy only | β13.3% | +96.4% |
| Quant + deterministic research | β1.8% | +13.4% |
| Quant + real LLM research | not yet measurable | not yet measurable |
Read this table carefully β it is easy to misread. The measured hybrid arm uses a deterministic research proxy (a momentum restatement), not an LLM. That proxy correlates with the quant signal while carrying more noise, so blending it diluted the ranking.
The real-LLM arm has not been measured. Its shadow comparison currently
reports INSUFFICIENT_SAMPLE, and forward collection is ongoing. Anyone telling
you whether LLM research adds alpha β positively or negatively β on evidence
like this is overclaiming. We are not going to.
What the LLM layer is observed to do today is intercept conflicts: when the quant and research signals disagree, it downgrades to reduce/no_action instead of averaging them away.
- The universe is 24 instruments per market. A Rank IC of +0.027 over 24 names is statistically thin. Treat these numbers as a demonstration that the protocol works, not as evidence that the model works. Widening the universe is the highest-priority open issue.
- Survivorship bias is present and flagged. The universe is today's liquid
names, not point-in-time index constituents. Every dataset carries
survivorship_bias_risk=true. - Market data comes from unofficial endpoints (Eastmoney / Sina / Tencent). They rate-limit, they change without notice, and their terms of service may restrict your use. There is no commercial data licence here. Check your own jurisdiction and the providers' terms before any non-personal use.
- LLM features are not in any model yet, by design, and will not be for months.
- No live broker path is validated. An Alpaca adapter exists; it has never traded real money.
citation_supportreturned 0.0 on every run until 2026-08 due to an evidence-id format mismatch. Fixed, with regression tests, but any LLM-fusion result recorded before that date understated the research signal.
git clone https://github.com/flysheep-ai/NoAlpha.git
cd NoAlpha
python -m venv .venv && source .venv/bin/activate
pip install -e ".[quant]" # core install omits the ML extras
cp .env.example .env # optional: add an LLM key to enable research
uvicorn noalpha.api:app --port 8000
cd frontend && npm install && npm run dev # console at http://127.0.0.1:5173Or docker compose up.
Configuration. Everything in .env.example is optional β with no keys the
system runs in deterministic mode and says so, rather than silently pretending an
LLM was involved. Secrets resolve in this order, highest first:
.cache/api/credentials.jsonβ set at runtime in the Settings UI, gitignored- exported environment variables β what CI and containers use
.envβ local development, gitignored
.env is never committed; .env.example documents every variable the code reads
and is enforced by a test.
noalpha demo leakage # the 13.3x inflation demo
python experiments/quant_phase_q.py --market cn # full 7-model walk-forward
python -m evals.runners.autonomous_replay # historical replay, closed loopsnoalpha trading create --initial-capital 100000 --markets CN,US
noalpha trading run-once
noalpha trading statuspytest tests/ -qIncludes 7 leakage-attack tests that try to sneak future information past the point-in-time guard, and hard invariant tests asserting that an executed proposal can never report as pending.
docs/adr/β 12 architecture decision records: why LangGraph, a knowledge graph, and a vector database were each rejected for nowdocs/audit/phase_q_report.mdβ the full walk-forward run behind the results tables above, including the leakage demodocs/audit/phase_q5_report.mdβ the LLM fusion work and why the LLM arm still reportsINSUFFICIENT_SAMPLEdocs/research/deep_dives/β source-level reads of TradingAgents, Qlib, Lean, NautilusTrader, OpenBB, Freqtrade and others
The highest-value contributions are the ones that make the measurement stricter, not the ones that make the numbers prettier:
- Widen the universe. 24 instruments per market is the biggest open weakness in every result table above β see limitations.
- Attack the point-in-time guard. If you can sneak future information past it, that is a bug report we want. Add it to the leakage-attack suite.
- Point-in-time index constituents, to retire the
survivorship_bias_riskflag.
See CONTRIBUTING.md for setup, test commands, and the measurement-integrity rules applied to pull requests. Please open an issue before a large PR.
Research and educational software. Not financial, investment, or trading advice. Simulated results are not indicative of real performance. Do not point this at real money.
MIT Β© flysheep-ai



