Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -35,3 +35,9 @@ outputs/
# GSD was removed on 2026-08-06. Keep its planning tree from silently
# reappearing at the repo root if a gsd-* command is ever run here again.
/.planning/

# public site build (scripts/site/publish.sh); only vercel.json and README are tracked
site/record/
site/static/
site/index.html
site/.vercel/
37 changes: 37 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,43 @@ autoMIL: F2-readiness framework refactor.
the shared tracker's does; the tracker accepts a `patience` of zero (a
one-epoch run, as the vendored stoppers already allowed) because a
`--hparams` override lands after the attempt is charged.
- **The dashboard is the project site, and the agent's record is on it.**
`automil viz start` serves a static site that reads one record layout
(`record/index.json`, per-run `graph.json`, `timeline.json`,
`sessions.json`, `agent_links.json`, `nodes/<id>.json`,
`sessions/<id>/turns/<k>.json`) built from the project's own files; the
same layout is written to disk by the new `automil viz export --out DIR`
(`--single-file` inlines it into one HTML page), so the hosted site, a
local server and an exported copy are the same frontend. Views: the
lineage tree with the primary-value chart and a node drawer that states
the keep/discard verdict with the graph's own helpers, the discovery
timeline (agent actions, experiment runs, best value on one axis), the
transcript reader (every turn, tool call, result and subagent, linked to
the nodes it created), the nodes table, the notes, the runs page and a
home page. The look is a white page with black rules, a visible column grid on
the hero, Manrope headlines and JetBrains Mono labels, teal for data only; runs
of tool-only turns fold into one line in the transcript. The hosted site can
connect to a user's own server through an
SSH tunnel: the server answers CORS only for `viz.cors_origins`
(default automil.org) and loopback origins, on the record routes and the
event stream. The tree is drawn in three dimensions by default (a top-down
layout on the page's paper, the kept lineage in teal; a flat lineage tree is one
click away); the old dark glow style is gone.
- **Session transcripts are stored by the framework.** `automil activity
ingest` copies the runtime's transcript and sidecar into
`automil/sessions/<session-id>/` on `SessionEnd`, and the hidden
`automil activity store-sessions [--root]` stores every journaled session
(the campaign launcher's exit trap calls it; `store_session_record.sh` is
removed). Open sessions are read from the runtime's own file and streamed
as they grow (`transcript_delta` frames on `/events`).
- **Validation-only, one enforcement point.** `viz/record_graph.py`
projects every node and result through `firewall.is_held_out_metric_key`,
no loader has a path under `archive/<node>/certify/`, run logs are served
for terminal nodes only and re-redacted, and secrets in tool results are
masked. `tests/viz/test_firewall_forged.py` plants a `test_auc` everywhere
and checks every payload and every opened path.
- `tests/viz/test_app_js_metrics_reader.py` (source-text assertions on the
old `app.js`) is replaced by behaviour tests on the served payloads.

- **Companion non-inferiority guard: a veto without a vote.** Single-metric
selection stays exactly as it was — the argmax is taken over
Expand Down
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ autoMIL gives coding agents the infrastructure to run experiments autonomously:
- Experiment tracking (directed tree, not flat log)
- Knowledge persistence (learnings.md)
- Result evaluation (val-firewall + Ladder keep/discard)
- 3D visualization (live dashboard)
- A site that shows the tree, the agent's transcript and every attempt's record, live or exported

</td>
</tr>
Expand Down Expand Up @@ -94,7 +94,7 @@ autoMIL gives coding agents the infrastructure to run experiments autonomously:
| **Trajectory recorder** | Per-submit JSONL using OpenTelemetry `gen_ai.*` keys with secret redaction (`sk-…`, `hf_…`, AWS keys) and bounded rotation (5 MB soft / 50 MB hard). |
| **Multi-GPU orchestrator** | Background daemon with bin packing, OOM detection, crash recovery, namespaced `running/<backend>/`. |
| **Experiment tree** | UCB-inspired scoring balances exploitation and exploration across branches; primary_value-dominance keep/discard gated by the Ladder keep-margin, on a **validation-only** primary_value scalar (the val-firewall). |
| **3D dashboard** | Interactive Three.js visualization with live SSE updates (`localhost:8420`). |
| **Dashboard and site** | `automil viz start` serves the project site at `localhost:8420`: the lineage tree with the value chart, the discovery timeline, the agent transcript (every tool call and result, linked to the nodes it created), every node's record and the notes, updated live. `automil viz export` writes the same site with the run recorded inside it. |
| **Persistent learnings** | Knowledge accumulates across sessions. Agents don't repeat mistakes. |
| **Setup validation** | `automil check` validates protected files, registry purity, backend directives, and `env.required` before experiments run. |

Expand Down Expand Up @@ -223,7 +223,7 @@ automil orchestrator start

# Terminal 2: visualization (optional)
tmux new -s viz
automil viz start # dashboard at localhost:8420
automil viz start # site at localhost:8420 (prints the ssh -L line when over SSH)
# Ctrl-b d to detach

# Terminal 3: agent loop
Expand All @@ -237,7 +237,7 @@ claude --dangerously-skip-permissions # autonomous mode, no permission prompts
```bash
automil status # quick summary
automil rank # top proposals
# Open http://localhost:8420 # 3D experiment tree
# Open http://localhost:8420 # tree, timeline, agent transcript, nodes, notes
```

---
Expand Down Expand Up @@ -404,7 +404,7 @@ automil trajectory record / export JSONL trajectory capture and r
# Loop + daemons
automil start-loop / stop-loop Control agent loop flag
automil orchestrator start / stop / status GPU scheduler daemon (best-fit bin packing)
automil viz start / stop / status 3D visualization dashboard at localhost:8420
automil viz start / stop / status / export Dashboard site at localhost:8420; export writes it as a static site
```

Run `automil <command> --help` for full flag listings.
Expand Down
20 changes: 13 additions & 7 deletions benchmarks/campaigns/preprint_130/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -395,15 +395,21 @@ only the fresh folds); a baseline whose retry cached every fold has no timing
and is submitted with `--cell <id> --e5-hours <five-fold hours>`. Nothing from a rehearsal set is mirrored,
frozen into the campaign selections, or certified.

**Session record.** Whatever the outcome, the job's exit copies the
runtime's own transcript of the cell's session,
**Session record.** The runtime's own transcript of the cell's session,
`~/.claude/projects/<cwd>/<session-id>.jsonl` (every user, assistant and
tool message) with its sidecar directory (subagent transcripts, fetched
tool results), from the submitter's home into `<cell root>/operator/session/`
(`store_session_record.sh <cell-root>`; every session the activity journal
saw, including one that failed before binding; idempotent; run it by hand
for a cell whose job predates it). The home copy is pruned by the runtime after
its cleanup period, so the cell root is the record, beside
tool results), is copied from the submitter's home into
`<cell root>/automil/sessions/<session-id>/` (`transcript.jsonl`,
`subagents/`, `record.json`). The framework's `SessionEnd` hook stores it as
the session ends, and the job's exit runs
`automil activity store-sessions --root <cell-root>` for every session the
activity journal saw (including one that failed before binding; idempotent;
run it by hand for a cell whose job predates it). Records made before this
layout sit at `<cell root>/operator/session/<session-id>.jsonl` and move with
`mkdir -p automil/sessions/<id> && mv operator/session/<id>.jsonl automil/sessions/<id>/transcript.jsonl && mv operator/session/<id>/* automil/sessions/<id>/`.
`automil viz start` in the cell root shows the session beside the tree. The
home copy is pruned by the runtime after its cleanup period, so the cell root
is the record, beside
`automil/.activity.jsonl` (session open, bind and end), `automil/graph.json`,
`automil/results.tsv`, `automil/learnings.md`, `automil/plan.md`,
`automil/orchestrator/orchestrator.log`, `operator/usage_before.txt`,
Expand Down
42 changes: 0 additions & 42 deletions benchmarks/scripts/slurm/store_session_record.sh

This file was deleted.

5 changes: 4 additions & 1 deletion benchmarks/scripts/slurm/submit_discovery_campaign.sh
Original file line number Diff line number Diff line change
Expand Up @@ -117,8 +117,11 @@ LOG="$LOG_DIR/${CELL}.log"
OPDIR="$RUNTIME/$CELL/operator"; mkdir -p "$OPDIR"
# The runtime's transcript of the session into the cell root, whatever the
# outcome (the home copy is pruned by the runtime after its cleanup period).
# The SessionEnd hook stores it as the session ends; this covers a runtime
# that was killed first. Same implementation, one storage layout:
# <cell root>/automil/sessions/<session-id>/.
store_session_record() { # cell
"$PROJECT_DIR/benchmarks/scripts/slurm/store_session_record.sh" "$RUNTIME/$1" >> "$LOG" 2>&1 \
uv run --frozen --no-sync automil activity store-sessions --root "$RUNTIME/$1" >> "$LOG" 2>&1 \
|| echo "[$1] session record not stored (see $LOG)"
}
trap 'store_session_record "$CELL"; normalize_cell_modes "$CELL"' EXIT
Expand Down
2 changes: 1 addition & 1 deletion benchmarks/src/autobench/campaign.py
Original file line number Diff line number Diff line change
Expand Up @@ -1258,7 +1258,7 @@ def materialize_discovery_cells(
json.dumps(cell, indent=2, sort_keys=True) + "\n"
)
(staging_adir / ".gitignore").write_text(
"graph.json\nresults.tsv\nresult.json\norchestrator/\ncells/\n"
"graph.json\nresults.tsv\nresult.json\norchestrator/\ncells/\nsessions/\n"
".activity.jsonl\n.activity.samples.json\n.activity.lock\n"
".automil_active\n.automil_worktrees/\n*.log\n*.pid\n"
)
Expand Down
4 changes: 2 additions & 2 deletions docs/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -282,7 +282,7 @@ automil status # quick summary
automil rank # top-ranked proposals (UCB)
automil cell list # cell budget state
automil orchestrator status # daemon health
# Open http://localhost:8420 for the 3D dashboard
# Open http://localhost:8420 for the dashboard (over SSH: ssh -N -L 8420:127.0.0.1:8420 user@host)
```

## How Experiments Run
Expand Down Expand Up @@ -397,7 +397,7 @@ test-selection bias — the rigor the framework exists to provide.
| `automil trajectory record / export` | JSONL trajectory capture + redacted export bundle |
| `automil start-loop` / `automil stop-loop` | Control agent loop flag |
| `automil orchestrator start / stop / status` | Manage GPU scheduler daemon |
| `automil viz start / stop / status` | Manage 3D dashboard |
| `automil viz start / stop / status / export` | The dashboard site (live), or exported as a static site |

Run `automil <command> --help` for the full flag listing.

Expand Down
31 changes: 31 additions & 0 deletions scripts/site/publish.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
#!/usr/bin/env bash
# Export recorded runs into site/ and deploy it to Vercel.
#
# Usage: scripts/site/publish.sh [--deploy] <cell-root>[:<run-id>[:<title>]] ...
#
# Each argument is a project or cell root (the directory holding automil/),
# optionally followed by the run id and title to show on the site. Cell roots
# copied from a cluster carry stamps in that host's zone: set SITE_TZ to its
# IANA name (default America/Vancouver, the campaign host). Without --deploy
# the site is only written, for a look with `python -m http.server -d site`.
set -euo pipefail
HERE="$(cd "$(dirname "$0")/../.." && pwd)"
SITE="$HERE/site"
TZ_NAME="${SITE_TZ:-America/Vancouver}"
deploy=0
if [ "${1:-}" = "--deploy" ]; then deploy=1; shift; fi
[ $# -gt 0 ] || { echo "usage: $0 [--deploy] <cell-root>[:<run-id>[:<title>]] ..." >&2; exit 2; }

rm -rf "$SITE/record" "$SITE/static" "$SITE/index.html"
for spec in "$@"; do
IFS=':' read -r root run_id title <<< "$spec"
args=(--out "$SITE" --tz "$TZ_NAME" --force)
[ -n "${run_id:-}" ] && args+=(--run-id "$run_id")
[ -n "${title:-}" ] && args+=(--title "$title")
(cd "$HERE" && uv run automil --project "$root" viz export "${args[@]}")
done
echo "site written to $SITE ($(du -sh "$SITE" | cut -f1))"
if [ $deploy = 1 ]; then
command -v vercel >/dev/null || { echo "vercel CLI not found: npm i -g vercel, then vercel login" >&2; exit 1; }
(cd "$SITE" && vercel deploy --prod)
fi
7 changes: 7 additions & 0 deletions site/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# The public site

`site/` is the deploy target for automil.org. Only `vercel.json` and this
file are tracked; the page, the static assets and the recorded runs are
written here by `scripts/site/publish.sh`, which exports each listed cell
root with `automil viz export` and deploys the directory with the Vercel CLI.
The site uses hash routes, so no rewrite rule is needed.
Loading
Loading