What it is

Glassbench connects finance agent systems to one platform, runs them on the same stock and date, and records what their agents read, argued and decided, so you can compare them and judge the method yourself instead of trusting a rating. It runs the official TradingAgents (v0.6.0), AI Hedge Fund (v2.5.0) and Vibe-Trading (v0.1.18) engines unmodified, each through one adapter, and your own system plugs in the same way.

• Front end, live and replay. One card per agent, grouped by stage, with what it is reading, writing and concluding; open a card for every LLM call, every tool call with its result and the reasoning stream. A stage strip and a timeline show who ran when. An AI Hedge Fund run gets one lane per investor agent and the earnings drift model, then the blend, the risk limits and the simulated fill. Every run replays step by step.
• One database of every run. Framework and version, provider and models, ticker, date, analysts, rating, entry, stop, target, horizon, tokens, tool calls, cost and flags, in SQLite. Filter on any of them, facet by model or variant, search the full text of every report and reasoning trace, export to CSV. This repository ships the author's database: 117 runs on 15 tickers, 101 from TradingAgents, 12 from AI Hedge Fund and 4 from Vibe-Trading.
• Benchmark models and frameworks. Pick the provider and type any model id; DeepSeek is the default, and OpenAI, Anthropic, Google, xAI, Qwen, GLM, MiniMax, Mistral, Kimi, Groq, OpenRouter and a local Ollama are one key away. Each run records the model asked for and the model actually served, so two models, or two frameworks, on the same stock and date sit side by side with their cost. Named variants change how the agents are fed (prompt wording, SEC EDGAR statements as filed, point-in-time valuation) without touching the engine (the valuation matched pair).
• Backtesting. Weekly grids of runs with a budget cap, next-open fills and trading costs, against buy-and-hold, a moving-average rule and a shuffled-rating placebo (the method).
• Broker connection. trade.py turns a finished run into one bracket order on an Interactive Brokers paper account, the trader's stop and the portfolio manager's target attached, and links the order back to the run.

A frameworks page keeps a reference sheet per framework: how it decides, what it reads, what the evidence says. Nothing here is a ready system to trade with; see Honest limits.

_{Tour of the app. Shown here: a stored TradingAgents run replayed about 75 times faster than it ran; a real run takes four to nine minutes.}

Not investment advice. Glassbench is a research and education tool. Every rating, price level and report in this repository and its database is output from language models, published so the method can be judged. None of it is a recommendation to buy, sell or hold any security, by the author or anyone else. Orders go to paper accounts only.

New

• Local models, Oct 2026. TradingAgents on two open-weight models served by llama.cpp on one desktop (Qwen3-30B-A3B and Qwen3.6-35B-A3B), 12 stocks each at no API cost, their ratings set beside DeepSeek's on the same stocks and date. Ratings, speed and every report in results/local-models, plus how to point Glassbench at a local server.
• Vibe-Trading, Oct 2026. A third framework: Vibe-Trading's own investment committee (bull, bear, risk officer, portfolio manager) or one research agent, with read-only tools and no shell, on DeepSeek or a local model. Sessions run in its own app can be imported.
• Harness Arena, Oct 2026. Five AI books on Alpaca paper accounts for two sessions: Claude Code and dsh crossed with Haiku and DeepSeek on one prompt, against TradingAgents. Code, daily reports and a luck test in arena/.

Screenshots

Runs, the record of every analysis	A run, with the committee board

Database, faceted over every run	Frameworks, the reference sheet

New run: any provider, any model id, your choice of agents	An agent opened: its calls, tools and output

A paper trade: the committee's decision becomes a bracket order at Interactive Brokers, target and stop attached

Frameworks

Framework	Status	Notes
TradingAgents 0.6.0	Connected	Engine runs untouched. Glassbench observes it through LangChain callbacks and reads its state; every node name and state key it depends on lives in one file, backend/deskapp/adapter.py, with tests. Optional variants (prompt wording, SEC EDGAR point-in-time valuation) are applied at runtime and never change the clone. Release 0.5.1 includes a fix and a bug report that came out of Glassbench runs (#1370, #1369). Release 0.5.1 also rewrote its own Hold guidance ("conflicting arguments alone are not a reason to Hold"), the idea the prompt wording variant tested, so that variant applies to 0.4.0 and 0.5.0 only; on later releases Glassbench reports it as unavailable instead of editing text that is no longer there. Release 0.6.0 runs each analyst as a graph of its own, with its own message history, and reads statements from SEC EDGAR by default; Glassbench streams those inner graphs, so every analyst's tokens, tool calls and report still land on its own card.
AI Hedge Fund 2.5.0	Connected	Runs untouched in its own clone and virtual environment, because it pins other LangChain versions; backend/drivers/aihf_driver.py drives one cycle and streams it as events. Release 2.5.0 splits a cycle into two steps, assess at a close and execute at the next completed close, and Glassbench sequences them for one stock and date. Data comes from Financial Datasets, the framework's own client, or from a free Yahoo Finance and SEC EDGAR route Glassbench adds, which is not native and is labelled on every run that uses it. Each investor agent and the earnings drift model is a lane, followed by the blend, the risk limits and the simulated execution. Every run keeps its own prompt cache, so a rerun asks the model again. AI Hedge Fund returns portfolio weights, and Glassbench maps the blended conviction to five tiers (0.50 or more Buy, 0.15 Overweight, above -0.15 Hold, above -0.50 Underweight, else Sell) so both frameworks share one table, and a run where most analysts abstained is REVIEW.
Vibe-Trading 0.1.18	Connected	Runs untouched in its own clone and virtual environment; backend/drivers/vt_driver.py builds its agents and streams them as events. By default Glassbench runs Vibe-Trading's own investment committee preset with its own prompts: bull and bear researchers in parallel, a chief risk officer, then a portfolio manager. Glassbench adds one paragraph to each prompt and asks the portfolio manager for a closing JSON block. A second mode runs one research agent on a fixed prompt. Research only: shell tools are off and every agent gets an allowlist of read-only data tools plus quantlib_call, Vibe-Trading's shell-free gate to its finance maths, so no file access, no backtests (they run generated code) and no broker tools. Each run gets its own Vibe-Trading home in the run folder. Vibe-Trading has no date input and no rating scale: runs dated in the past are flagged as not point in time, and the closing block maps to the five tiers (a stated rating, else long Overweight, short Underweight, wait or hedge Hold). Sessions run in Vibe-Trading's own app can be imported, tagged as imported and kept apart from controlled runs.

Every table's Framework column shows the framework and version only (TradingAgents 0.6.0, AI Hedge Fund 2.5.0, Vibe-Trading 0.1.18). A build that contains changes on top of the release is labelled adjusted, with the changes listed in the tooltip.

Quick start

You need Python 3.11 or newer, Node 22 or newer, Git, and an API key for one LLM provider, or a model running on your own machine (see Local models, under step 6). DeepSeek is the default and costs about $0.06 per full run; eleven other providers and local Ollama are listed in .env.example.

1. Clone Glassbench and the engine.

git clone https://github.com/davidalmeida90/glassbench.git
cd glassbench
git clone --branch v0.6.0 --depth 1 https://github.com/TauricResearch/TradingAgents.git

The engine is expected at ./TradingAgents (or a sibling ../TradingAgents, or wherever GLASSBENCH_ENGINE_DIR points).

2. Python environment.

py -3.11 -m venv .venv            # Mac or Linux: python3 -m venv .venv
.\.venv\Scripts\Activate.ps1      #                source .venv/bin/activate
pip install -r requirements.txt
pip install -e .\TradingAgents

3. Keys. Copy .env.example to .env and paste your key after the equals sign (DEEPSEEK_API_KEY, or the variable of the provider you will pick). Only the names listed there are read, and values never reach the browser.

4. Optional: AI Hedge Fund.

git clone https://github.com/virattt/ai-hedge-fund.git
cd ai-hedge-fund
py -3.11 -m venv .venv            # its own environment: it pins other LangChain versions
.\.venv\Scripts\python.exe -m pip install -e .
.\.venv\Scripts\python.exe -m pip install yfinance==1.7.0   # only for the free data route
cd ..

It is found at ./ai-hedge-fund, a sibling ../ai-hedge-fund, or wherever GLASSBENCH_AIHF_DIR points. Its own prices and fundamentals come from Financial Datasets: add FINANCIAL_DATASETS_API_KEY to .env. Its key is free and its data prepaid, about $0.02 a request, and a run takes four to six. Its entry plan returns the last twelve months only, and each investor agent needs four filed quarters, so on that plan pick a recent date. Without that key, pick Yahoo Finance (free) under Data: Glassbench's own adapter (Yahoo Finance prices, SEC EDGAR statements), not native to AI Hedge Fund, so results might not be ideal, and labelled on every run that uses it. Then choose AI Hedge Fund in New run: a library strategy or your own analysts, one model for all of them, and the data.

5. Optional: Vibe-Trading.

git clone https://github.com/HKUDS/Vibe-Trading.git
cd Vibe-Trading
py -3.11 -m venv .venv            # its own environment
.\.venv\Scripts\python.exe -m pip install --require-hashes -r requirements-lock.txt
.\.venv\Scripts\python.exe -m pip install --no-deps -e .
cd ..

It is found at ./Vibe-Trading, a sibling ../Vibe-Trading, or wherever GLASSBENCH_VT_DIR points. On Windows, keep Glassbench in a short folder or turn on long paths: a committee's working files sit about 135 characters below the run folder. It runs on DeepSeek or on a local model (below). Choose Vibe-Trading in New run, then its investment committee or one research agent, and a model. Sessions you ran in Vibe-Trading's own app come in through Import from Vibe-Trading on the Vibe-Trading tab, which reads ~/.vibe-trading or wherever GLASSBENCH_VT_HOME points.

6. Start.

.\glassbench.ps1                  # Mac or Linux: ./glassbench.sh

The first start builds the frontend, then the app is at http://127.0.0.1:8765. Press New run, pick a ticker, a date, the analysts, the provider and the two models, and watch the committee work. A full four-analyst run takes about eight minutes and costs about six cents on DeepSeek.

Headless, without the browser:

cd backend
python -m deskapp run NVDA --analysts market,news,fundamentals

Local models, no API key. Glassbench also runs on a model served from your own machine: Ollama, LM Studio, llama.cpp's llama-server or any server with an OpenAI compatible /v1 endpoint. Set OLLAMA_BASE_URL in .env to that address (Ollama's default, http://localhost:11434/v1, needs nothing), choose Ollama (local) in New run and type the model id the server serves as both models. Tokens, calls and reports are recorded as for any provider, at a cost of $0. Setup, speed and a comparison with DeepSeek are in results/local-models.

The runs database

data/desk.db is SQLite, and this repository ships the author's own database as a courtesy: 117 runs on 15 tickers (NVDA, AAPL, MSFT, INTC, PFE, JPM, KO, MCD, O, T, TSLA, XOM, AMZN, GOOGL, META), September and October 2026.

• TradingAgents, 88 runs on 0.4.0, 0.5.0, 0.5.1 and 0.6.0, with DeepSeek V4 Flash as the quick model and V4 Pro as the deep model. They include the two Microsoft runs from the video that disagree on the same day, the NVIDIA run that became a paper order, a weekly backtest pilot, matched pairs of variants, and a batch of twin runs on eight stocks (25 September 2026) in which seven runs failed when the DeepSeek balance ran out; they stay in the database as failed. Each run's reports, tool outputs and final state are under data/runs/<run_id>/.
• TradingAgents on local models, 13 runs on 0.6.0, 28 October 2026, served by llama.cpp on one desktop at no API cost, memory off: 8 on Qwen3.6-35B-A3B (NVIDIA, Apple, Microsoft, Amazon, Alphabet, Meta, Tesla, JPMorgan), 3 on Qwen3-30B-A3B-Instruct-2507 and 2 on the dense Qwen3.8-27B. A run took about 10 minutes on the two mixture-of-experts models and about 50 on the dense one. Setup and speed are in results/local-models.
• AI Hedge Fund, 12 runs with DeepSeek V4.1 Flash. Nine on 2.4.0: four complete cycles on four companies and four strategies (NVIDIA, Apple, JPMorgan and Tesla, late August to mid September 2026), one more on NVIDIA in October, and four that show the limits of the entry data plan: two dates older than its twelve months fail, and on two others the investor agents had fewer than four filed quarters and abstained. Three on 2.5.0, all NVIDIA on 8 October 2026 with Deep Value: one on Financial Datasets (Hold) and two on the free route (Overweight both times), labelled as free data. One draw per route at most, so the ratings show that the routes can disagree, not why.
• Vibe-Trading, 4 runs on 0.1.18, all NVIDIA on 28 October 2026. One research agent on DeepSeek V4 Flash (Buy). Three investment committees on local Qwen3.6-35B-A3B: the first two failed and stay in the database as failed (a reply that ran to 84k tokens with queued requests timing out behind it, then a researcher whose context filled), and they set the local settings the driver now uses (replies capped at 4,096 tokens, one agent at a time, at most 10 steps each). The third finished at Underweight in 28 minutes at no API cost; the same committee on DeepSeek, outside this database, said Buy.

AI Hedge Fund runs publish derived outputs only: each analyst's signal and reasoning, the blend, the weights and the simulated fill (record.json). Financial Datasets responses and the prompts that quote them stay out of the repository, because its licence allows sharing only what is derived from the data; the event log records each data call as a row count and a date span. Runs on the free route are published the same way.

-- ratings by ticker
SELECT ticker, rating, COUNT(*) FROM runs WHERE status = 'finished' GROUP BY 1, 2 ORDER BY 1;

-- what the portfolio manager wrote, full text search
SELECT run_id, agent, snippet(event_text, 4, '[', ']', '...', 12) FROM event_text WHERE event_text MATCH 'valuation' LIMIT 20;

The Runs and Database pages export any filtered view to CSV. Every finished run stays in the database, by design: the disagreements are the point.

Where	What
data/desk.db	the runs table, every event, and a full-text index of the logs
data/runs/<run_id>/reports/	analyst reports, both debates, trader proposal, risk debate, final decision
data/runs/<run_id>/tool_outputs/	the full output of every tool call the agents made
data/runs/<run_id>/state.json	the final state of the graph
data/runs/<run_id>/record.json	an AI Hedge Fund cycle: every signal, the blend, the risk limits, the simulated orders and fills
data/aihf_private/<run_id>/	AI Hedge Fund's raw data responses and prompts, local only and ignored by Git
data/orders.jsonl	paper orders sent by trade.py, each linked to its run
state/trading_memory.md	the engine's decision memory (created on first run)

Data and keys

Source	Used for	Key
DeepSeek (default), OpenAI, Anthropic, Google, xAI, Qwen, GLM, MiniMax, Mistral, Kimi, Groq, OpenRouter, or a local Ollama	the agents, chosen per run	that provider's key in .env (DEEPSEEK_API_KEY, OPENAI_API_KEY, ...); Ollama needs none
Yahoo Finance	prices, indicators, news; prices and earnings surprises on AI Hedge Fund's free route	none
SEC EDGAR	statements as filed (TradingAgents' default since 0.6.0), the point-in-time valuation variant, and AI Hedge Fund's free route	SEC_EDGAR_EMAIL (the SEC asks callers to identify themselves)
FRED	macro series	FRED_API_KEY, free
StockTwits, Reddit	the sentiment analyst	none
Financial Datasets	AI Hedge Fund's own prices, fundamentals and earnings	FINANCIAL_DATASETS_API_KEY, prepaid credits; not needed on the free route
TypeSafe	Jev as AI Hedge Fund's model, and TradingAgents' screening of social posts (since 0.5.1)	TYPESAFE_API_KEY, optional
Interactive Brokers, Alpaca	paper orders and connection checks	TWS or IB Gateway logged into a paper account; ALPACA_API_KEY and ALPACA_SECRET_KEY for Alpaca paper

Everything the engine downloads goes into state/cache/. Keys are read once, from .env, by the backend process; the browser gets their names and lengths, never their values.

A trade at the broker

py -3 trade.py NVDA MSFT              # run the twelve agents on each stock, then trade
py -3 trade.py NVDA MSFT --reuse      # replay each stock's latest finished run, then trade
py -3 trade.py NVDA --dry-run         # everything except sending the orders

Long only, one decision, fixed size: Buy or Overweight becomes one marketable limit order for $10,000 worth with the trader's stop and the portfolio manager's target attached as a bracket; Hold does nothing; Underweight or Sell closes what is held. Paper ports and paper accounts only, one working order per stock, size capped, and a y/N confirmation before anything is sent. It needs Glassbench running and Trader Workstation or IB Gateway logged into a paper account.

What a run records

Twelve agents: four data analysts (market, sentiment, news, fundamentals) with tools, a bull and a bear researcher, a research manager, a trader, three risk debaters and a portfolio manager. Cheap fast models do the reading and the debating; the expensive model is called only by the two managers. Every tool returns data dated on or before the run date, which is what makes a backtest possible.

An AI Hedge Fund run records each analyst's lane (the data it fetched, as counts and dates, and its one LLM call), the blend, the risk limits and the simulated fill at the next close.

The full mechanism, agent by agent, with the tool calls from a real run: docs/how-a-run-works.md. The backtest method and the pilot: docs/backtest-method.md. What serving point-in-time valuation changes in what the agents write, measured on 24 runs: docs/evaluation-valuation-pair.md. Notes on how to evaluate these systems, and which published results hold up: docs/backtest-research.md.

Honest limits

• The TradingAgents paper reports one backtest on three stocks over three months (January to March 2024). An independent test over longer windows landed below buy-and-hold. A Sharpe of 8 over one quarter says the idea deserves a test, not that it works.
• Two runs on the same ticker and date can differ. The database holds twin runs two minutes apart with opposite ratings. Measure how often that happens before trusting any single rating.
• The backtest pilot in this repository (AAPL and NVDA, weekly, 2025 to 2026) had the agents' trader levels lose money while buy-and-hold gained, and the ratings alone never traded. Trading costs, next-open fills and a shuffled-rating placebo are in the simulator so the comparison stays fair.
• AI Hedge Fund's valuation fields come from the latest filed period, so its investor agents can see a price from months before the run date, and they abstain when fewer than four quarters are on file.
• Glassbench's free data route rebuilds AI Hedge Fund's inputs from SEC filings and Yahoo. It is not the framework's own data, the two can differ, and a run on one route says nothing about a run on the other until the same cells have been run on both.
• The order in the video was sent to a paper account. No real money moved. Nothing in this repository is investment advice.

Layout

glassbench/
├── glassbench.ps1, glassbench.sh   launchers (build the frontend once, then serve)
├── trade.py                        one bracket order per stock at an IB paper account
├── backend/deskapp/
│   ├── adapter.py      every TradingAgents node name, state key and parser (the one file to fix after an engine update)
│   ├── capture.py      LangChain callbacks and engine warnings turned into events
│   ├── runner.py       runs the graph with streaming, memory and reports; hands AI Hedge Fund and Vibe-Trading runs to their adapters
│   ├── aihf.py         AI Hedge Fund: what the clone offers, the subprocess runner, costs, the rating mapping
│   ├── vibe.py         Vibe-Trading: committee and single-agent layouts, the subprocess runner, the closing block's rating
│   ├── vibe_import.py  sessions and swarm runs from Vibe-Trading's own app, imported on their original timeline
│   ├── runconfig.py    one engine config per run (the engine keeps a process-wide one)
│   ├── variants.py     runtime variants: prompt wording, SEC EDGAR statements and valuation
│   ├── edgar.py        point-in-time fundamentals from SEC EDGAR
│   ├── backtest.py     weekly grids of runs with a budget cap
│   ├── simulate.py     the simulator: fills, costs, baselines, placebo
│   ├── paired.py       matched-pair reading of what agents wrote
│   ├── brokers.py      Alpaca and Interactive Brokers paper connection checks
│   ├── frameworks.py   the reference sheets on the Frameworks page
│   ├── store.py        SQLite, events, full-text index
│   ├── api.py          REST and Server-Sent Events (replay, then live)
│   └── settings.py     paths, models, price table, engine config
├── backend/drivers/    aihf_driver.py and vt_driver.py: one AI Hedge Fund cycle or one Vibe-Trading analysis inside its own environment, printed as events
├── backend/tests/      adapter, backtest, simulator, variants, run config, AI Hedge Fund, Vibe-Trading
├── frontend/src/       React and TypeScript (Vite)
├── data/               the runs database and the run files
├── results/            local-models/: TradingAgents on local models, ratings, speed, reports
├── arena/              Harness Arena: five AI books on Alpaca paper, code and daily reports
└── docs/               how a run works, backtest method, the valuation matched pair, research notes, screenshots

After updating the engine, run the adapter tests; a failing test names what changed upstream, and adapter.py is where to fix it:

cd backend
python -m unittest discover tests

Development

cd frontend && npm run dev        # Vite on http://127.0.0.1:5173, /api proxied to the backend on 8765
cd backend && python -m unittest discover tests

Engine internals live in one place, backend/deskapp/adapter.py; capture.py turns LangChain callbacks into numbered events; store.py keeps them in SQLite with a full-text index; api.py replays them over Server-Sent Events and then switches to live. Anything that changes how the agents think is a named variant in variants.py, applied per run through runconfig.py, so the engine clone is never edited.

Roadmap

• Both data routes on the same cells. AI Hedge Fund's free route (Yahoo Finance prices, SEC EDGAR statements) now runs; next is running the same stocks and dates on both routes, several times each, to measure how much of a rating moves with the data rather than with the model's draw.
• Transparency pack. A grounding check of each report against the data it pulled, rerun dispersion (how often twin runs disagree), and a compare view.
• Scheduled sessions. A watchlist run on a clock, with rules that turn ratings into orders only when several runs agree.
• More brokers and guardrails. Alpaca paper orders next to Interactive Brokers, a kill switch, a cost budget.

Contributing

Issues and pull requests are welcome, in particular more framework adapters, more brokers, and evaluation methods. Keep engines untouched: anything Glassbench needs from a framework goes through its adapter, and anything that changes how agents think is a named variant applied at runtime.

Credits and license

Glassbench is by David Arias, CFA, released under the Apache License 2.0. It runs on top of TradingAgents by Tauric Research (Apache 2.0), which is not bundled here; see the paper by Yijia Xiao, Edward Sun, Di Luo and Wei Wang. It also runs AI Hedge Fund by Virat Singh (MIT), also not bundled. The video that walks through all of this: Open Source AI Trading Agents on DeepSeek: Setup to a Broker Order.

Disclaimer

Glassbench is software for research and education. It is not investment advice, a recommendation, or an offer to buy or sell any security, and it is not a trading system.

• Ratings, price levels, reports and debates in the published runs are output from language models. They are wrong often, two runs on the same stock and date can disagree, and they are published so the method can be examined, never so they can be followed.
• Backtests and paper trades describe the past on a handful of decisions. They say nothing reliable about future returns.
• The software connects to paper broker accounts only. Anyone who adapts it to trade real money does so at their own risk and under their own responsibility.
• Views expressed here are the author's own and do not represent any employer or client. Glassbench is independent of Tauric Research, the TradingAgents project and AI Hedge Fund.
• The software is provided "as is", without warranty of any kind, under the Apache License 2.0.

Citation

If you use the runs database or the simulator in research, cite the TradingAgents paper and link this repository:

@misc{xiao2025tradingagents,
  title  = {TradingAgents: Multi-Agents LLM Financial Trading Framework},
  author = {Yijia Xiao and Edward Sun and Di Luo and Wei Wang},
  year   = {2025},
  eprint = {2412.20138},
  archivePrefix = {arXiv},
  url    = {https://arxiv.org/abs/2412.20138}
}