diff --git a/README.md b/README.md
index 6950c21..aca4c7d 100644
--- a/README.md
+++ b/README.md
@@ -116,7 +116,7 @@ frontier_agent/ generic loop, scheduling, registries, AgentBus, observers
plugins/tools/ web, shell, file, sandbox, and team tool implementations
workflows/ ReAct and Agent Team pipelines, profiles, prompts, observers
apodex/ terminal CLI/TUI, approvals, sessions, traces, and Docker path
-benchmarks/ public benchmark harness plus standalone FrontierSearchBench
+benchmarks/ public harness plus bundled FrontierSearchBench/FrontierChallenge
```
More detail: [framework architecture](docs/framework.md),
diff --git a/benchmarks/README.md b/benchmarks/README.md
index 0b5b6d4..148a4d5 100644
--- a/benchmarks/README.md
+++ b/benchmarks/README.md
@@ -9,13 +9,13 @@ The benchmark tree has three deliberately separate areas:
benchmarks such as GDPval, HLE, OfficeQA, BrowseComp, and FrontierScience,
together with their registry, runner, judges, dataset tooling, and local
result directories.
-- [`frontier_search_bench/`](frontier_search_bench/) is the independently
- maintained [FrontierSearchBench](https://github.com/ApodexAI/frontier-search-bench)
- source tree. It is kept at the top level so it remains visible and can be
- synced to its future standalone open-source repository.
-- [`frontierchallenge/`](frontierchallenge/) contains FrontierChallenge's
- standalone Harbor runtime, 97-task registry, taxonomy, image recipes,
- documentation, and Hugging Face dataset setup flow.
+- [`frontier_search_bench/`](frontier_search_bench/) contains
+ [FrontierSearchBench](frontier_search_bench/): 41 verifiable deep-search
+ queries, their official scorers, and batch evaluation tooling. Answer
+ collection is integrated through the adapter under [`public/`](public/).
+- [`frontierchallenge/`](frontierchallenge/) contains
+ [FrontierChallenge](frontierchallenge/)'s Harbor runtime, 97-task registry,
+ taxonomy, image recipes, documentation, and Hugging Face dataset setup flow.
The public evaluation harness runs one question per Python subprocess. Runs are
independently reproducible, resumable, and protected from a single hung task by
@@ -40,7 +40,7 @@ Each dataset registers its default workflow and scoring implementation. Passing
| `frontier_science_olympiad` | FrontierScience | `stateful-react-agent` | LLM judge |
| `deepsearchqa` | DeepSearchQA | `stateful-react-agent` | LLM judge |
| `widesearch` | WideSearch | `stateful-react-agent` | Structural F1 |
-| `frontier_search` | FrontierSearchBench | `stateful-react-agent` | Official external batch scorer |
+| `frontier_search` | FrontierSearchBench | `stateful-react-agent` | Bundled post-collection batch scorer |
| `officeqa` | OfficeQA | `stateful-react-agent` | Official deterministic reward |
| `officeqa_full` | OfficeQA-Full | `stateful-react-agent` | Official deterministic reward |
| `gdpval` | GDPval | `stateful-react-agent` | Deterministic deliverable validation |
@@ -108,8 +108,8 @@ benchmarks/
│ ├── scripts/ dataset download and standardization
│ ├── datasets/ local source data (gitignored)
│ └── results/ local run artifacts (gitignored)
-├── frontier_search_bench/ standalone benchmark source and official scorers
-└── frontierchallenge/ standalone scientific-workflow benchmark runtime
+├── frontier_search_bench/ FrontierSearchBench queries and official scorers
+└── frontierchallenge/ FrontierChallenge scientific-workflow runtime
```
## Add a benchmark
diff --git a/benchmarks/frontier_search_bench/README.md b/benchmarks/frontier_search_bench/README.md
index 69b6867..dc74de7 100644
--- a/benchmarks/frontier_search_bench/README.md
+++ b/benchmarks/frontier_search_bench/README.md
@@ -2,7 +2,7 @@
Apodex's benchmark for deep search / deep research. The goal is to compare AI platforms side by side on challenging tasks that are long-horizon, retrieval-heavy, and verifiable.
-The repository holds two things:
+This directory holds two things:
1. **Verifiable benchmark queries** (`queries/verifiable.json`) — 41 evaluation inputs whose answers are unique or programmatically checkable (numbers, entities, enumerable sets, …).
2. **Evaluation scripts** (`eval/verifiable/`) — auto-scoring of model answers (41/41 implemented).
@@ -13,7 +13,12 @@ The repository holds two things:
queries/verifiable.json ──► (external collection: platform answers → unified JSON) ──► eval/verifiable/
```
-Answer collection does not happen in this repository; the eval scripts only consume the **unified JSON** interface below.
+The benchmark implementation in this directory does not collect answers; its
+eval scripts consume the **unified JSON** interface below. FrontierAgent's
+collection adapter lives in
+[`../public/families/frontier_search.py`](../public/families/frontier_search.py),
+and the complete collect/export/score workflow is documented in
+[`../../docs/eval-frontier-search.md`](../../docs/eval-frontier-search.md).
**Query file** (`queries/verifiable.json`, 41 items) — each entry looks like:
@@ -47,10 +52,10 @@ The eval scripts read `report_content` and fall back to `response`. This is the
> python3 -c 'import json,pathlib;p=pathlib.Path;src=json.loads(p("queries/verifiable.json").read_text(encoding="utf-8"));p("queries/answer_template.json").write_text(json.dumps([{"id":e["id"],"query":e["query"],"report_content":"","response":""} for e in src],ensure_ascii=False,indent=2)+"\n",encoding="utf-8")'
> ```
-## Repository layout
+## Directory layout
```
-frontier-search-bench/
+benchmarks/frontier_search_bench/
├── queries/
│ └── verifiable.json # 41 items
└── eval/
@@ -63,7 +68,10 @@ frontier-search-bench/
Requires **Python 3.10+**.
+From the FrontierAgent repository root:
+
```bash
+cd benchmarks/frontier_search_bench
pip install -r eval/verifiable/requirements.txt
cp eval/verifiable/.env.example eval/verifiable/.env # fill in OPENROUTER_API_KEY
```
@@ -72,7 +80,9 @@ See [`eval/verifiable/README.md`](eval/verifiable/README.md) for details.
## Quickstart
-Prerequisite: unified JSON answer files prepared per the data contract above.
+Prerequisite: unified JSON answer files prepared per the data contract above,
+with the working directory set to `benchmarks/frontier_search_bench/` as shown
+in the environment setup.
**1. Score a single query** (full instructions in [`eval/verifiable/README.md`](eval/verifiable/README.md)):
diff --git a/benchmarks/frontier_search_bench/UPSTREAM.md b/benchmarks/frontier_search_bench/UPSTREAM.md
index a933f7d..f50ce03 100644
--- a/benchmarks/frontier_search_bench/UPSTREAM.md
+++ b/benchmarks/frontier_search_bench/UPSTREAM.md
@@ -1,29 +1,17 @@
-# Upstream provenance
+# Maintenance notes
-This directory is imported from
-[`ApodexAI/frontier-search-bench`](https://github.com/ApodexAI/frontier-search-bench)
-with Git subtree.
+This directory is maintained directly in FrontierAgent at
+`benchmarks/frontier_search_bench`. It contains the canonical query set and
+official scorers; it is not synchronized from a separate public repository.
-- Imported upstream commit: `0a5323b9823f8ee05486bbae11ca96999e5d5af9`
-- FrontierAgent prefix: `benchmarks/frontier_search_bench`
-- Import mode: squashed subtree
-
-This directory documents itself as if answer collection happened elsewhere,
-because upstream has no collector. In FrontierAgent it does: the adapter in
-`benchmarks/public/families/frontier_search.py` feeds the shared subprocess
-runner, and [`docs/eval-frontier-search.md`](../../docs/eval-frontier-search.md)
-owns the collect / export / score workflow. Read that instead of inferring a FrontierAgent
-workflow from the upstream README.
-
-Keep FrontierAgent adapters outside this directory when possible. Pull a future
-upstream release from the repository root with:
-
-```bash
-git subtree pull \
- --prefix=benchmarks/frontier_search_bench \
- https://github.com/ApodexAI/frontier-search-bench.git main --squash
-```
+Keep FrontierAgent-specific collection adapters outside this directory when
+possible. The adapter in
+[`../public/families/frontier_search.py`](../public/families/frontier_search.py)
+feeds the shared subprocess runner, while
+[`../../docs/eval-frontier-search.md`](../../docs/eval-frontier-search.md) owns
+the collect/export/score workflow.
The official scorers contain ground truth. Do not expose this directory to the
-agent process during a scored collection run; see
-`docs/eval-frontier-search.md` for the isolation requirement.
+agent process during a scored collection run; see the
+[FrontierSearchBench evaluation guide](../../docs/eval-frontier-search.md) for
+the isolation requirement.
diff --git a/benchmarks/frontier_search_bench/eval/verifiable/README.md b/benchmarks/frontier_search_bench/eval/verifiable/README.md
index be793e4..4dab285 100644
--- a/benchmarks/frontier_search_bench/eval/verifiable/README.md
+++ b/benchmarks/frontier_search_bench/eval/verifiable/README.md
@@ -1,6 +1,7 @@
# Verifiable Eval — auto-scoring scripts
-Programmatic scoring for the 41 queries in `queries/verifiable.json` at the repository root.
+Programmatic scoring for the 41 queries in `queries/verifiable.json` at the
+FrontierSearchBench directory root.
## Directory layout
@@ -88,7 +89,9 @@ cp .env.example .env
# edit .env and fill in OPENROUTER_API_KEY=sk-...
```
-`.env` may live in `eval/verifiable/`, its parent directory, or the repository root — all three locations are loaded automatically.
+Place `.env` at `eval/verifiable/.env`, the location consistently supported by
+the batch runner and all query scorers. Alternatively, export
+`OPENROUTER_API_KEY` and `OPENROUTER_BASE_URL` in the environment.
The code reads two environment variables: `OPENROUTER_API_KEY` (required) and `OPENROUTER_BASE_URL` (optional, default `https://openrouter.ai/api/v1`). Models are called through the OpenAI SDK.
diff --git a/benchmarks/frontier_search_bench/eval/verifiable/scorers/query_15/auto_scorer.py b/benchmarks/frontier_search_bench/eval/verifiable/scorers/query_15/auto_scorer.py
index e289742..390b582 100644
--- a/benchmarks/frontier_search_bench/eval/verifiable/scorers/query_15/auto_scorer.py
+++ b/benchmarks/frontier_search_bench/eval/verifiable/scorers/query_15/auto_scorer.py
@@ -72,7 +72,7 @@ def get_client():
base_url=base_url,
api_key=api_key,
default_headers={
- "HTTP-Referer": "https://github.com/ApodexAI/frontier-search-bench",
+ "HTTP-Referer": "https://github.com/ApodexAI/FrontierAgent",
"X-Title": "frontier-search-bench",
},
)
diff --git a/benchmarks/frontier_search_bench/eval/verifiable/scorers/query_38/auto_scorer.py b/benchmarks/frontier_search_bench/eval/verifiable/scorers/query_38/auto_scorer.py
index b733d84..381320c 100644
--- a/benchmarks/frontier_search_bench/eval/verifiable/scorers/query_38/auto_scorer.py
+++ b/benchmarks/frontier_search_bench/eval/verifiable/scorers/query_38/auto_scorer.py
@@ -231,7 +231,7 @@ def get_client():
base_url=base_url,
api_key=api_key,
default_headers={
- "HTTP-Referer": "https://github.com/ApodexAI/frontier-search-bench",
+ "HTTP-Referer": "https://github.com/ApodexAI/FrontierAgent",
"X-Title": "frontier-search-bench",
},
)
diff --git a/benchmarks/frontierchallenge/site/README.md b/benchmarks/frontierchallenge/site/README.md
index b2e9a55..40e0697 100644
--- a/benchmarks/frontierchallenge/site/README.md
+++ b/benchmarks/frontierchallenge/site/README.md
@@ -1,14 +1,14 @@
# FrontierChallenge website
This directory contains the dependency-free static website for the benchmark.
-It lives in the public FrontierChallenge repository so website content, task
-documentation, and leaderboard links can evolve together.
+It lives in FrontierAgent alongside FrontierChallenge's task documentation and
+leaderboard links so they can evolve together.
Published preview:
## Preview locally
-From the repository root, run:
+From `benchmarks/frontierchallenge/`, run:
```sh
python3 -m http.server 8000 --directory site
@@ -16,15 +16,17 @@ python3 -m http.server 8000 --directory site
Then visit . Stop the server with `Ctrl-C`.
-## Build for hosting
+## Build the Worker bundle
```sh
python3 site/build_site.py
```
-The generated `site/dist/` directory is intentionally ignored by Git and is
-recreated for each hosting build. GitHub Pages deployment is defined in
-`.github/workflows/pages.yml`.
+The generated Worker module under `site/dist/` is intentionally ignored by Git
+and recreated for each bundle. The separate GitHub Pages deployment is defined
+in the repository-level
+[`frontierchallenge-pages.yml`](../../../.github/workflows/frontierchallenge-pages.yml)
+workflow.
## Paper figures
diff --git a/benchmarks/public/README.md b/benchmarks/public/README.md
index 949fafe..a6ee0ad 100644
--- a/benchmarks/public/README.md
+++ b/benchmarks/public/README.md
@@ -5,11 +5,11 @@ public benchmarks, including BrowseComp, HLE, OfficeQA, GDPval, and
FrontierScience. It owns the registry, dataset adapters, judges, subprocess
runner, download/standardization scripts, and gitignored local artifacts.
-The independently maintained FrontierSearchBench source is intentionally kept
-outside this package at [`../frontier_search_bench/`](../frontier_search_bench/).
-The adapter in `families/frontier_search.py` lets the shared runner collect its
-answers without mixing FrontierAgent-specific runtime code into that source
-tree.
+The bundled [FrontierSearchBench](../frontier_search_bench/) queries and scorers
+are kept in a sibling directory. The adapter in
+`families/frontier_search.py` lets the shared runner collect its answers while
+keeping FrontierAgent-specific runtime code out of the benchmark
+implementation.
Run a one-question smoke evaluation after installing the eval dependencies and
placing the selected dataset under `datasets/`:
diff --git a/benchmarks/public/runner/score_frontier_search.py b/benchmarks/public/runner/score_frontier_search.py
index 8744e72..e5a022a 100644
--- a/benchmarks/public/runner/score_frontier_search.py
+++ b/benchmarks/public/runner/score_frontier_search.py
@@ -41,7 +41,7 @@ def _scorer_env() -> dict[str, str]:
env["OPENROUTER_BASE_URL"] = env.get(url_var) or _DEFAULT_BASE_URL
return env
- # No credentials anywhere. Leave the key empty so the upstream scorer's
+ # No credentials anywhere. Leave the key empty so the bundled scorer's
# own "OPENROUTER_API_KEY not set" check reports it.
env["OPENROUTER_API_KEY"] = ""
env.setdefault("OPENROUTER_BASE_URL", _DEFAULT_BASE_URL)
diff --git a/docs/eval-frontier-search.md b/docs/eval-frontier-search.md
index afd21cd..38ffe7f 100644
--- a/docs/eval-frontier-search.md
+++ b/docs/eval-frontier-search.md
@@ -12,19 +12,19 @@ around the checkout. That is why it has its own page — [`eval.md`](eval.md) ow
everything the benchmarks share (installation, judge preflight, datasets, the
runner's options, results), and this page owns only what is specific here.
-The benchmark source lives at
-[`benchmarks/frontier_search_bench/`](../benchmarks/frontier_search_bench/) as a
-squashed Git subtree of
-[`ApodexAI/frontier-search-bench`](https://github.com/ApodexAI/frontier-search-bench);
-see [`UPSTREAM.md`](../benchmarks/frontier_search_bench/UPSTREAM.md) for the
-import provenance and the `git subtree pull` command. FrontierAgent's adapter is
-deliberately outside that tree, in `benchmarks/public/families/frontier_search.py`,
-so the subtree stays syncable. The upstream README states that answer collection
-does not happen in its repository — inside FrontierAgent it does, through the
-shared subprocess runner described below.
+The canonical benchmark implementation lives at
+[`benchmarks/frontier_search_bench/`](../benchmarks/frontier_search_bench/).
+FrontierAgent's collection adapter is deliberately kept outside that directory,
+in
+[`benchmarks/public/families/frontier_search.py`](../benchmarks/public/families/frontier_search.py),
+so the query/scorer implementation remains separate from the shared runtime.
+See its [maintenance notes](../benchmarks/frontier_search_bench/UPSTREAM.md) for
+the ownership and isolation boundaries. The benchmark implementation consumes
+unified JSON answer files; inside FrontierAgent, the shared subprocess runner
+collects and exports those answers as described below.
No dataset download is needed. The 41 queries and all 41 official scorers are
-bundled with the subtree.
+bundled with FrontierAgent.
## Evaluation isolation requirement
@@ -46,7 +46,7 @@ and prints a warning into the run's own output; it is not evidence of isolation.
## Credentials
-The wrapper maps `JUDGE_API_KEY` and `JUDGE_BASE_URL` onto the upstream scorer's
+The wrapper maps `JUDGE_API_KEY` and `JUDGE_BASE_URL` onto the scorer's
`OPENROUTER_API_KEY` and `OPENROUTER_BASE_URL`, always as a pair from one
provider — set `OPENROUTER_*` directly and the `JUDGE_*` values are left alone.
@@ -81,7 +81,7 @@ interrupted collection continues where it stopped.
## 2. Export
-Export the run into the upstream unified JSON contract. For a multi-run
+Export the run into the benchmark's unified JSON contract. For a multi-run
evaluation, export each `run_` separately as one model/run input.
```bash
@@ -98,8 +98,9 @@ of 41 would report a headline score for the 12 easiest. Finish the run, or pass
## 3. Score
Run all 41 official scorers. The wrapper reuses `JUDGE_API_KEY` and
-`JUDGE_BASE_URL`, executes a temporary copy of the upstream tree, and preserves
-aggregate plus per-query artifacts under the requested result directory.
+`JUDGE_BASE_URL`, executes a temporary copy of the benchmark implementation,
+and preserves aggregate plus per-query artifacts under the requested result
+directory.
```bash
uv run python -m benchmarks.public.runner.score_frontier_search \
@@ -128,9 +129,9 @@ an explicit step.
## Scoring internals
-Upstream's own documentation is the reference for what the scorers do:
+The bundled scorer documentation is the reference for what the scorers do:
[`eval/verifiable/README.md`](../benchmarks/frontier_search_bench/eval/verifiable/README.md)
covers the CLI, the three-stage pipeline, normalization, and the meaning of
coverage. The unified JSON answer contract it consumes — and which
-`export_frontier_search` produces — is documented in the subtree's
+`export_frontier_search` produces — is documented in the benchmark's
[`README.md`](../benchmarks/frontier_search_bench/README.md).
diff --git a/docs/framework.md b/docs/framework.md
index 45fcd59..7179cb5 100644
--- a/docs/framework.md
+++ b/docs/framework.md
@@ -10,7 +10,7 @@ plugins, and benchmark evaluation. The framework layer has no dependency on
frontier_agent/ generic loop, scheduling, registries, AgentBus, observers
plugins/tools/ tool implementations and sandbox policy
workflows/ pipeline specs, profiles, prompts, workflow-owned observers
-benchmarks/ public benchmark harness plus standalone FrontierSearchBench
+benchmarks/ public harness plus bundled FrontierSearchBench/FrontierChallenge
```
## Runtime flow