guruagents-reproduction

A bounded reproduction of the Warren Buffett backtest reported in GuruAgents: Emulating Wise Investors with Prompt-Guided LLM Agents (Kim et al., arXiv:2510.01664, 2025).

This repository ports the relevant backtest code path from the paper's companion repository to a clean Python module, runs it on the bundled data published in the same repository, and reports the resulting performance metrics side by side with the values printed in Table 1 of the paper. No external data sources are introduced and no methodological modifications are made.

Result

Faithful Python port of the backtest code path in 04_multi_agent_backtesting.ipynb of yejining99/GuruAgents (the relevant logic is identical in 03_backtesting.ipynb), executed on the bundled data/nasdaq100_ohlcv.csv and the seven Buffett quarterly portfolios in results/buffett_agent/ from the same commit:

metric	reproduction	paper Table 1 (Buffett row)	residual (repro − paper)	paper column heading
Cumulative return (%)	50.66	42.23	+8.43	"CAGR"
Annualized return — 252-day geometric (%)	19.38	26.03	-6.65	"mean (ann.)"
Sharpe ratio (annualized)	0.99	1.40	-0.41	"Sharpe (ann.)"
Maximum drawdown (%)	-23.02	-22.34	-0.68	"MDD"

The reproduction window covers 583 trading days (2024-01-01 .. 2025-08-12) and uses all seven published Buffett portfolios chained by the end-of-period compounding rule that the upstream code implements. Maximum drawdown reproduces close to parity; the headline gap is concentrated in the cumulative-return and annualized-return columns, which sit on opposite sides of the published values.

Scope

This repository tests one claim:

The values reported for the Warren Buffett row in arXiv:2510.01664, Table 1, are reproducible from the paper's companion repository under the backtest code path that repository ships.

The result is that they are not reproducible for the cumulative-return and annualized-return columns; the residuals are listed above. The repository does not test alternative agents, does not propose a corrected Table 1, and does not assert a cause for the residuals. It provides the pipeline so that any reader can reach the same numbers from a clean checkout in a few minutes.

A separate observation that emerged during the audit, and that the paper authors have acknowledged in correspondence, is that the values placed in Table 1's "CAGR" column are computed as cumulative return rather than as a compound annual growth rate; Section 3.4 of the paper expands "CAGR" as "Cumulative Annual Growth Rate", which mirrors the same conflation. This labeling observation is documented in notebooks/03_label_audit.py but it is not the central reproduction claim, because relabeling alone does not close the residuals listed above.

How to reproduce

Requires Python 3.11 or 3.12 and uv. Initial run also requires internet access to fetch ~90 MB of upstream data.

git clone <this-repo>
cd guruagents-reproduction
uv sync
uv run python scripts/fetch_paper_artifacts.py
uv run guruagents-reproduce

The fetch step downloads the seven Buffett portfolio CSVs and nasdaq100_ohlcv.csv (~90 MB) from the upstream repository at fetch time, against the commit pinned in src/guruagents_repro/sources.py, and verifies each against the SHA-256 digests committed there. Nothing under data/raw/ is committed to this repository.

For the test suite:

uv run pytest

For the executed notebooks (the .py files under notebooks/ are the canonical source; jupytext renders them as .ipynb with outputs preserved):

uv run jupytext --to notebook --execute notebooks/*.py

Repository layout

guruagents-reproduction/
├── data/
│   ├── PROVENANCE.md                  # SHA-256 of every upstream file we read
│   └── raw/                           # gitignored; populated by fetch_paper_artifacts.py
│       ├── data/nasdaq100_ohlcv.csv
│       └── results/buffett_agent/buffett_portfolio_*.csv
├── src/guruagents_repro/
│   ├── sources.py                     # upstream URL + commit pin + SHA-256 manifest
│   ├── io.py                          # download + SHA-256 verification + load helpers
│   ├── schema.py                      # column expectations
│   ├── weights.py                     # 1:1 port of weight normalization
│   ├── pipeline.py                    # 1:1 port of calculate_agent_returns
│   ├── metrics.py                     # 1:1 port of calculate_performance_metrics
│   ├── report.py                      # comparison frame + paper Table 1 row
│   └── cli.py                         # `guruagents-reproduce` entry point
├── scripts/
│   └── fetch_paper_artifacts.py       # download + verify all upstream files
├── tests/                             # 29 tests: provenance, weights, invariants, targets
└── notebooks/                         # five jupytext .py walkthroughs
    ├── 00_provenance.py
    ├── 01_pipeline_replication.py
    ├── 02_results_comparison.py
    ├── 03_label_audit.py
    └── 04_summary.py

Methodological notes

The pipeline port preserves four behaviors of the upstream code that a reader might otherwise have to discover by hand:

• Weight normalization is applied at portfolio-load time with the rule weight = weight / sum(weight) * 100 (the published portfolio CSVs sum to 105–131% across the seven Buffett quarters; the upstream code rescales them to 100 before backtesting).

• Investment windows run from analysis_end + 1 day through analysis_end + 90 days, a closed interval spanning 90 calendar dates (89 days between the endpoints). Successive windows are non-overlapping; the number of skipped calendar dates between windows varies with quarter length and weekend layout.

• Price column selection uses [col for col in columns if 'CLOSE' in col.upper()][0], which resolves to CLOSE_ given the column order of nasdaq100_ohlcv.csv. This is the close column distinct from the explicitly dividend-adjusted DIV_ADJ_CLOSE that follows it in the CSV. A sibling utility get_price_data defined in the same notebook prefers DIV_ADJ_CLOSE, but it is not invoked from the backtest path.

• Metric formulas for cumulative return and annualized return are the upstream definitions, reproduced verbatim:

  cumulative_pct  = (final_value - 1) * 100
  annualized_pct  = (final_value ** (252 / N_trading_days) - 1) * 100

The port deliberately does not apply transaction costs (the calculate_turnover_and_costs utility exists in the upstream notebook but is not invoked from the code path that produces Table 1's numbers) and does not substitute alternative price sources or weighting schemes.

Upstream license

At the time of this audit the upstream repository carries no LICENSE file and the GitHub API reports license: null. We therefore do not redistribute any upstream file in this repository; everything under data/raw/ is fetched on demand and gitignored. See NOTICE for the exact list of files fetched at runtime.

Citation

If you cite this reproduction, please also cite the paper it audits:

Kim, Y. et al. GuruAgents: Emulating Wise Investors with Prompt-Guided LLM Agents. arXiv:2510.01664 (2025).