Skip to content

program.md

Latest commit

 

History

History
208 lines (157 loc) · 9.55 KB

program.md

File metadata and controls

208 lines (157 loc) · 9.55 KB
Raw

autoquant

Setup

  1. Inspect git status --short --branch. In a human-started run, confirm a unique run tag and create arena/<tag> from master; never reuse a tag. In an arena worktree, the driver has already created the branch: do not switch it.

  2. Run uv sync and uv run python setup_data.py --verify. It must print DATA OK.

  3. Read strategy.py and evaluate.py fully. Read data/NOISE_FLOOR.json and data/BASELINE.json. If lineage.json is absent it is seeded from the baseline on first score.

  4. Run uv run python run.py. It prints the new runs/<n> directory only after trusted code has atomically written score.json. Confirm the score matches data/BASELINE.json to 1e-8.

  5. Create results.tsv if absent, using this TAB-separated header exactly:

    commit score mean_sharpe std_sharpe max_dd turnover status description

The setup baseline is not an experiment: never append it to results.tsv. Experiment 1 starts with the first new hypothesis and pre-run commit.

Experimentation

You CAN:

  • Edit anything in strategy.py: features, signal logic, hyperparameters, and helper functions.
  • Read and plot TRAIN-period data to form hypotheses.
  • Read evaluate.py to understand exactly how the evaluator scores you.
  • Read lineage.json, recursion_metrics.json, and runs/<n>/score.json including the structured comparison and decision blocks.

You CANNOT:

  • Modify evaluate.py, run.py, setup_data.py, program.md, lineage.json by hand, or anything in data/. Protected changes fail their hashes or the arena audit and disqualify the run. lineage.json is advanced only by the evaluator after a trial.
  • Install packages or import outside numpy, pandas, scipy, numba, math, itertools, and dataclasses.
  • Read, print, serialize, or otherwise inspect raw VALID rows. Strategy code receives a frame only so it can compute signals. Treat that capability as evaluator-owned, not an invitation to exfiltrate observations.
  • Look at LOCKBOX. It is unavailable without the arena's one-shot token.

Your goal is to maximize the official champion score in score.json / lineage.json:

score = mean(fold_sharpes) - 0.5 * std(fold_sharpes)

Returns are net of immutable fees, spread, square-root impact, and historical funding. Hard risk and causality gates apply before acceptance.

Simplicity criterion. A +0.01 improvement that adds 40 lines of special-case logic is probably not worth it. A +0.01 improvement from deleting code is worth keeping.

AUTOQUANT v2 — two ratchets and an archive

Search progress and official validation are intentionally separate.

candidate
   ↓
search acceptance → lineage archive → next mutations
   ↓
official promotion test → confirmed champion
   ↓ once (arena)
sealed multi-regime final evaluation (lockbox)

Working parent (search acceptance)

A candidate may become the new working parent when the evaluator reports decision.search_accepted: true. The default policy is:

  • gates PASS
  • score improves on the working parent by more than materiality_margin (0.02 — an explicit margin, not “noise”)
  • at least 4 of 6 folds improve versus the working parent
  • 2× and 3× cost Sharpes are not materially worse (cost_deltas / cost policy in evaluate)

Search accepts are logged with status search. They advance recursion even when the move is too small for official promotion.

Official champion (promotion)

A candidate becomes the official champion when decision.official_promoted: true (status keep). The official bar stays strong for backward compatibility:

  • score must clear the champion by the official margin (at least the calibrated noise_floor, and never below materiality adjusted for prior promotion attempts)
  • at least 5 of 6 folds improve versus the official champion
  • cost sensitivity is not materially worse

Only keep rows are confirmed promotions. The dashboard draws official progress as a solid line and working-parent progress as a dashed line.

Lineage archive (four active parents)

lineage.json keeps four archive slots so a rejected official candidate can still remain a useful parent on another dimension:

Slot Preference
best_overall highest score
most_stable lowest fold Sharpe dispersion among competitive scores
lowest_cost_sensitivity best worst-of 2×/3× cost Sharpes
most_distinct lowest return correlation with other archived lineages

Status archive means the commit was retained for at least one slot without becoming the working parent. You may switch the mutation base by restoring an archive commit’s strategy.py (see loop below). Unconditional git reset --hard HEAD~1 is not the default when a candidate claims an archive slot or search accept.

Structured feedback (read this every trial)

After every run, read runs/<n>/score.json fields:

  • comparison.score_delta — candidate versus working parent
  • comparison.fold_deltas — six fold Sharpe deltas
  • comparison.cost_deltas — 1× / 2× / 3× Sharpe deltas
  • comparison.drawdown_delta, comparison.turnover_delta
  • comparison.return_correlation — versus parent when returns exist
  • comparison.archive_correlations — versus each archive slot
  • comparison.gate_headroom — distance to hard gates
  • decision.status / decision.reasons / decision.archive_claims
  • decision.reset_to — full commit hash to restore on discard/tie

That feedback is the recursive signal. Do not invent a private score from VALID rows.

Overfitting rules

  • Official promotion uses the calibrated noise_floor as a statistical bar, not a search step size. Search uses materiality_margin instead.
  • If a change helps exactly one fold and hurts elsewhere, discard it.
  • Prefer fewer parameters when score and fold agreement are comparable.
  • State an economic mechanism. Update HYPOTHESIS whenever the signal changes. Say who is on the other side, why they keep trading, and what would make the effect decay.
  • Never tune toward a particular fold. You will ultimately be checked on data you cannot open.
  • When stuck in parameter sweeps, switch to another archive lineage (most_distinct or most_stable) or a structurally different mechanism.

Output format

After every run, read only the runs/<n>/score.json path printed by run.py. Do not parse stdout for metrics. Do not read run.log unless the run crashed. On a crash, inspect only tail -n 50 runs/<n>/run.log; evaluator sanitization strips instruction-like text first.

Logging results

Append one TAB-separated row to results.tsv after every attempt:

commit score mean_sharpe std_sharpe max_dd turnover status description

status is one of:

status meaning
keep official champion promotion
search working-parent search accept
archive retained in lineage archive only
discard rejected; restore working parent
tie positive but inside official bar / failed search gates
crash no score.json
gated hard gate failure
tampered protection failure

The arena may add dq to its public state. Use 0.000000 for crashes and -999.000000 for gate/tamper failures. Log every experiment. The complete trace is the benchmark.

Copy status from score.json → decision.status whenever that field exists. Do not second-guess the evaluator.

The experiment loop

  1. Read git log --oneline -20, the tail of results.tsv, and lineage.json. Know the official champion, working parent, and archive tips.

  2. Form one hypothesis. Prefer mutating the working parent; periodically branch from a non-best archive slot when diversity collapses (high archive correlations).

  3. Edit only strategy.py, including HYPOTHESIS.

  4. Commit before running: git add strategy.py && git commit -m "<one-line description>".

  5. Run with a five-minute timeout: uv run python run.py. Record the printed run directory.

  6. Read that directory's score.json; never grep a metric from logs.

  7. No score.json means crash. Read only the sanitized last 50 log lines. Fix trivial syntax or import errors, but abandon a deeper failure after two tries.

  8. Append one result row using decision.status. Record the commit with git rev-parse HEAD (full hash).

  9. Act on decision.action:

    • keep_commit (keep / search / archive): leave the commit on the branch. lineage.json is already updated by the evaluator.
    • reset: run git reset --hard <decision.reset_to> (the working-parent commit). This reset is authorized only inside the disposable autonomous competitor worktree. Never reset master.

  10. Optional lineage switch: to mutate a different archive tip, restore that commit’s genome without destroying history of other keeps:

    git checkout <archive_commit> -- strategy.py && git commit -m "switch lineage to <slot>"

    Then continue the loop from step 2. Do not invent commits; use hashes from lineage.json.

The arena's experiment budget is authoritative. Do not create extra unlogged trials, change the data, or evade a timeout.

NEVER STOP

Do not ask “should I keep going?” The human may be asleep. Continue until the arena reaches its experiment budget or the human interrupts the process.

If ideas run thin, re-read the evaluator, comparison blocks on near-miss trials, and lineage.json. Inspect TRAIN funding and open-interest columns. Prefer a structurally different economic hypothesis to another tiny parameter sweep. Near-misses may combine across archive lineages, but added complexity must earn its keep under the search or official bar.