From 6cfde3ef48ef815bd3b1f5e69e60324323b8a93f Mon Sep 17 00:00:00 2001 From: Yue Deng Date: Tue, 25 Aug 2026 17:35:34 +0800 Subject: [PATCH 1/2] docs: fix benchmark repository references --- README.md | 2 +- benchmarks/README.md | 20 +++++----- benchmarks/frontier_search_bench/README.md | 20 +++++++--- benchmarks/frontier_search_bench/UPSTREAM.md | 38 +++++++------------ .../eval/verifiable/README.md | 7 +++- .../scorers/query_15/auto_scorer.py | 2 +- .../scorers/query_38/auto_scorer.py | 2 +- benchmarks/frontierchallenge/site/README.md | 16 ++++---- benchmarks/public/README.md | 10 ++--- .../public/runner/score_frontier_search.py | 2 +- docs/eval-frontier-search.md | 35 ++++++++--------- docs/framework.md | 2 +- 12 files changed, 80 insertions(+), 76 deletions(-) diff --git a/README.md b/README.md index 6950c21..aca4c7d 100644 --- a/README.md +++ b/README.md @@ -116,7 +116,7 @@ frontier_agent/ generic loop, scheduling, registries, AgentBus, observers plugins/tools/ web, shell, file, sandbox, and team tool implementations workflows/ ReAct and Agent Team pipelines, profiles, prompts, observers apodex/ terminal CLI/TUI, approvals, sessions, traces, and Docker path -benchmarks/ public benchmark harness plus standalone FrontierSearchBench +benchmarks/ public harness plus bundled FrontierSearchBench/FrontierChallenge ``` More detail: [framework architecture](docs/framework.md), diff --git a/benchmarks/README.md b/benchmarks/README.md index 0b5b6d4..148a4d5 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -9,13 +9,13 @@ The benchmark tree has three deliberately separate areas: benchmarks such as GDPval, HLE, OfficeQA, BrowseComp, and FrontierScience, together with their registry, runner, judges, dataset tooling, and local result directories. -- [`frontier_search_bench/`](frontier_search_bench/) is the independently - maintained [FrontierSearchBench](https://github.com/ApodexAI/frontier-search-bench) - source tree. It is kept at the top level so it remains visible and can be - synced to its future standalone open-source repository. -- [`frontierchallenge/`](frontierchallenge/) contains FrontierChallenge's - standalone Harbor runtime, 97-task registry, taxonomy, image recipes, - documentation, and Hugging Face dataset setup flow. +- [`frontier_search_bench/`](frontier_search_bench/) contains + [FrontierSearchBench](frontier_search_bench/): 41 verifiable deep-search + queries, their official scorers, and batch evaluation tooling. Answer + collection is integrated through the adapter under [`public/`](public/). +- [`frontierchallenge/`](frontierchallenge/) contains + [FrontierChallenge](frontierchallenge/)'s Harbor runtime, 97-task registry, + taxonomy, image recipes, documentation, and Hugging Face dataset setup flow. The public evaluation harness runs one question per Python subprocess. Runs are independently reproducible, resumable, and protected from a single hung task by @@ -40,7 +40,7 @@ Each dataset registers its default workflow and scoring implementation. Passing | `frontier_science_olympiad` | FrontierScience | `stateful-react-agent` | LLM judge | | `deepsearchqa` | DeepSearchQA | `stateful-react-agent` | LLM judge | | `widesearch` | WideSearch | `stateful-react-agent` | Structural F1 | -| `frontier_search` | FrontierSearchBench | `stateful-react-agent` | Official external batch scorer | +| `frontier_search` | FrontierSearchBench | `stateful-react-agent` | Bundled post-collection batch scorer | | `officeqa` | OfficeQA | `stateful-react-agent` | Official deterministic reward | | `officeqa_full` | OfficeQA-Full | `stateful-react-agent` | Official deterministic reward | | `gdpval` | GDPval | `stateful-react-agent` | Deterministic deliverable validation | @@ -108,8 +108,8 @@ benchmarks/ │ ├── scripts/ dataset download and standardization │ ├── datasets/ local source data (gitignored) │ └── results/ local run artifacts (gitignored) -├── frontier_search_bench/ standalone benchmark source and official scorers -└── frontierchallenge/ standalone scientific-workflow benchmark runtime +├── frontier_search_bench/ FrontierSearchBench queries and official scorers +└── frontierchallenge/ FrontierChallenge scientific-workflow runtime ``` ## Add a benchmark diff --git a/benchmarks/frontier_search_bench/README.md b/benchmarks/frontier_search_bench/README.md index 69b6867..dc74de7 100644 --- a/benchmarks/frontier_search_bench/README.md +++ b/benchmarks/frontier_search_bench/README.md @@ -2,7 +2,7 @@ Apodex's benchmark for deep search / deep research. The goal is to compare AI platforms side by side on challenging tasks that are long-horizon, retrieval-heavy, and verifiable. -The repository holds two things: +This directory holds two things: 1. **Verifiable benchmark queries** (`queries/verifiable.json`) — 41 evaluation inputs whose answers are unique or programmatically checkable (numbers, entities, enumerable sets, …). 2. **Evaluation scripts** (`eval/verifiable/`) — auto-scoring of model answers (41/41 implemented). @@ -13,7 +13,12 @@ The repository holds two things: queries/verifiable.json ──► (external collection: platform answers → unified JSON) ──► eval/verifiable/ ``` -Answer collection does not happen in this repository; the eval scripts only consume the **unified JSON** interface below. +The benchmark implementation in this directory does not collect answers; its +eval scripts consume the **unified JSON** interface below. FrontierAgent's +collection adapter lives in +[`../public/families/frontier_search.py`](../public/families/frontier_search.py), +and the complete collect/export/score workflow is documented in +[`../../docs/eval-frontier-search.md`](../../docs/eval-frontier-search.md). **Query file** (`queries/verifiable.json`, 41 items) — each entry looks like: @@ -47,10 +52,10 @@ The eval scripts read `report_content` and fall back to `response`. This is the > python3 -c 'import json,pathlib;p=pathlib.Path;src=json.loads(p("queries/verifiable.json").read_text(encoding="utf-8"));p("queries/answer_template.json").write_text(json.dumps([{"id":e["id"],"query":e["query"],"report_content":"","response":""} for e in src],ensure_ascii=False,indent=2)+"\n",encoding="utf-8")' > ``` -## Repository layout +## Directory layout ``` -frontier-search-bench/ +benchmarks/frontier_search_bench/ ├── queries/ │ └── verifiable.json # 41 items └── eval/ @@ -63,7 +68,10 @@ frontier-search-bench/ Requires **Python 3.10+**. +From the FrontierAgent repository root: + ```bash +cd benchmarks/frontier_search_bench pip install -r eval/verifiable/requirements.txt cp eval/verifiable/.env.example eval/verifiable/.env # fill in OPENROUTER_API_KEY ``` @@ -72,7 +80,9 @@ See [`eval/verifiable/README.md`](eval/verifiable/README.md) for details. ## Quickstart -Prerequisite: unified JSON answer files prepared per the data contract above. +Prerequisite: unified JSON answer files prepared per the data contract above, +with the working directory set to `benchmarks/frontier_search_bench/` as shown +in the environment setup. **1. Score a single query** (full instructions in [`eval/verifiable/README.md`](eval/verifiable/README.md)): diff --git a/benchmarks/frontier_search_bench/UPSTREAM.md b/benchmarks/frontier_search_bench/UPSTREAM.md index a933f7d..f50ce03 100644 --- a/benchmarks/frontier_search_bench/UPSTREAM.md +++ b/benchmarks/frontier_search_bench/UPSTREAM.md @@ -1,29 +1,17 @@ -# Upstream provenance +# Maintenance notes -This directory is imported from -[`ApodexAI/frontier-search-bench`](https://github.com/ApodexAI/frontier-search-bench) -with Git subtree. +This directory is maintained directly in FrontierAgent at +`benchmarks/frontier_search_bench`. It contains the canonical query set and +official scorers; it is not synchronized from a separate public repository. -- Imported upstream commit: `0a5323b9823f8ee05486bbae11ca96999e5d5af9` -- FrontierAgent prefix: `benchmarks/frontier_search_bench` -- Import mode: squashed subtree - -This directory documents itself as if answer collection happened elsewhere, -because upstream has no collector. In FrontierAgent it does: the adapter in -`benchmarks/public/families/frontier_search.py` feeds the shared subprocess -runner, and [`docs/eval-frontier-search.md`](../../docs/eval-frontier-search.md) -owns the collect / export / score workflow. Read that instead of inferring a FrontierAgent -workflow from the upstream README. - -Keep FrontierAgent adapters outside this directory when possible. Pull a future -upstream release from the repository root with: - -```bash -git subtree pull \ - --prefix=benchmarks/frontier_search_bench \ - https://github.com/ApodexAI/frontier-search-bench.git main --squash -``` +Keep FrontierAgent-specific collection adapters outside this directory when +possible. The adapter in +[`../public/families/frontier_search.py`](../public/families/frontier_search.py) +feeds the shared subprocess runner, while +[`../../docs/eval-frontier-search.md`](../../docs/eval-frontier-search.md) owns +the collect/export/score workflow. The official scorers contain ground truth. Do not expose this directory to the -agent process during a scored collection run; see -`docs/eval-frontier-search.md` for the isolation requirement. +agent process during a scored collection run; see the +[FrontierSearchBench evaluation guide](../../docs/eval-frontier-search.md) for +the isolation requirement. diff --git a/benchmarks/frontier_search_bench/eval/verifiable/README.md b/benchmarks/frontier_search_bench/eval/verifiable/README.md index be793e4..f931ef1 100644 --- a/benchmarks/frontier_search_bench/eval/verifiable/README.md +++ b/benchmarks/frontier_search_bench/eval/verifiable/README.md @@ -1,6 +1,7 @@ # Verifiable Eval — auto-scoring scripts -Programmatic scoring for the 41 queries in `queries/verifiable.json` at the repository root. +Programmatic scoring for the 41 queries in `queries/verifiable.json` at the +FrontierSearchBench directory root. ## Directory layout @@ -88,7 +89,9 @@ cp .env.example .env # edit .env and fill in OPENROUTER_API_KEY=sk-... ``` -`.env` may live in `eval/verifiable/`, its parent directory, or the repository root — all three locations are loaded automatically. +`.env` may live in `eval/verifiable/`, its parent directory, or the +FrontierSearchBench directory root — all three locations are loaded +automatically. The code reads two environment variables: `OPENROUTER_API_KEY` (required) and `OPENROUTER_BASE_URL` (optional, default `https://openrouter.ai/api/v1`). Models are called through the OpenAI SDK. diff --git a/benchmarks/frontier_search_bench/eval/verifiable/scorers/query_15/auto_scorer.py b/benchmarks/frontier_search_bench/eval/verifiable/scorers/query_15/auto_scorer.py index e289742..390b582 100644 --- a/benchmarks/frontier_search_bench/eval/verifiable/scorers/query_15/auto_scorer.py +++ b/benchmarks/frontier_search_bench/eval/verifiable/scorers/query_15/auto_scorer.py @@ -72,7 +72,7 @@ def get_client(): base_url=base_url, api_key=api_key, default_headers={ - "HTTP-Referer": "https://github.com/ApodexAI/frontier-search-bench", + "HTTP-Referer": "https://github.com/ApodexAI/FrontierAgent", "X-Title": "frontier-search-bench", }, ) diff --git a/benchmarks/frontier_search_bench/eval/verifiable/scorers/query_38/auto_scorer.py b/benchmarks/frontier_search_bench/eval/verifiable/scorers/query_38/auto_scorer.py index b733d84..381320c 100644 --- a/benchmarks/frontier_search_bench/eval/verifiable/scorers/query_38/auto_scorer.py +++ b/benchmarks/frontier_search_bench/eval/verifiable/scorers/query_38/auto_scorer.py @@ -231,7 +231,7 @@ def get_client(): base_url=base_url, api_key=api_key, default_headers={ - "HTTP-Referer": "https://github.com/ApodexAI/frontier-search-bench", + "HTTP-Referer": "https://github.com/ApodexAI/FrontierAgent", "X-Title": "frontier-search-bench", }, ) diff --git a/benchmarks/frontierchallenge/site/README.md b/benchmarks/frontierchallenge/site/README.md index b2e9a55..40e0697 100644 --- a/benchmarks/frontierchallenge/site/README.md +++ b/benchmarks/frontierchallenge/site/README.md @@ -1,14 +1,14 @@ # FrontierChallenge website This directory contains the dependency-free static website for the benchmark. -It lives in the public FrontierChallenge repository so website content, task -documentation, and leaderboard links can evolve together. +It lives in FrontierAgent alongside FrontierChallenge's task documentation and +leaderboard links so they can evolve together. Published preview: ## Preview locally -From the repository root, run: +From `benchmarks/frontierchallenge/`, run: ```sh python3 -m http.server 8000 --directory site @@ -16,15 +16,17 @@ python3 -m http.server 8000 --directory site Then visit . Stop the server with `Ctrl-C`. -## Build for hosting +## Build the Worker bundle ```sh python3 site/build_site.py ``` -The generated `site/dist/` directory is intentionally ignored by Git and is -recreated for each hosting build. GitHub Pages deployment is defined in -`.github/workflows/pages.yml`. +The generated Worker module under `site/dist/` is intentionally ignored by Git +and recreated for each bundle. The separate GitHub Pages deployment is defined +in the repository-level +[`frontierchallenge-pages.yml`](../../../.github/workflows/frontierchallenge-pages.yml) +workflow. ## Paper figures diff --git a/benchmarks/public/README.md b/benchmarks/public/README.md index 949fafe..a6ee0ad 100644 --- a/benchmarks/public/README.md +++ b/benchmarks/public/README.md @@ -5,11 +5,11 @@ public benchmarks, including BrowseComp, HLE, OfficeQA, GDPval, and FrontierScience. It owns the registry, dataset adapters, judges, subprocess runner, download/standardization scripts, and gitignored local artifacts. -The independently maintained FrontierSearchBench source is intentionally kept -outside this package at [`../frontier_search_bench/`](../frontier_search_bench/). -The adapter in `families/frontier_search.py` lets the shared runner collect its -answers without mixing FrontierAgent-specific runtime code into that source -tree. +The bundled [FrontierSearchBench](../frontier_search_bench/) queries and scorers +are kept in a sibling directory. The adapter in +`families/frontier_search.py` lets the shared runner collect its answers while +keeping FrontierAgent-specific runtime code out of the benchmark +implementation. Run a one-question smoke evaluation after installing the eval dependencies and placing the selected dataset under `datasets/`: diff --git a/benchmarks/public/runner/score_frontier_search.py b/benchmarks/public/runner/score_frontier_search.py index 8744e72..e5a022a 100644 --- a/benchmarks/public/runner/score_frontier_search.py +++ b/benchmarks/public/runner/score_frontier_search.py @@ -41,7 +41,7 @@ def _scorer_env() -> dict[str, str]: env["OPENROUTER_BASE_URL"] = env.get(url_var) or _DEFAULT_BASE_URL return env - # No credentials anywhere. Leave the key empty so the upstream scorer's + # No credentials anywhere. Leave the key empty so the bundled scorer's # own "OPENROUTER_API_KEY not set" check reports it. env["OPENROUTER_API_KEY"] = "" env.setdefault("OPENROUTER_BASE_URL", _DEFAULT_BASE_URL) diff --git a/docs/eval-frontier-search.md b/docs/eval-frontier-search.md index afd21cd..38ffe7f 100644 --- a/docs/eval-frontier-search.md +++ b/docs/eval-frontier-search.md @@ -12,19 +12,19 @@ around the checkout. That is why it has its own page — [`eval.md`](eval.md) ow everything the benchmarks share (installation, judge preflight, datasets, the runner's options, results), and this page owns only what is specific here. -The benchmark source lives at -[`benchmarks/frontier_search_bench/`](../benchmarks/frontier_search_bench/) as a -squashed Git subtree of -[`ApodexAI/frontier-search-bench`](https://github.com/ApodexAI/frontier-search-bench); -see [`UPSTREAM.md`](../benchmarks/frontier_search_bench/UPSTREAM.md) for the -import provenance and the `git subtree pull` command. FrontierAgent's adapter is -deliberately outside that tree, in `benchmarks/public/families/frontier_search.py`, -so the subtree stays syncable. The upstream README states that answer collection -does not happen in its repository — inside FrontierAgent it does, through the -shared subprocess runner described below. +The canonical benchmark implementation lives at +[`benchmarks/frontier_search_bench/`](../benchmarks/frontier_search_bench/). +FrontierAgent's collection adapter is deliberately kept outside that directory, +in +[`benchmarks/public/families/frontier_search.py`](../benchmarks/public/families/frontier_search.py), +so the query/scorer implementation remains separate from the shared runtime. +See its [maintenance notes](../benchmarks/frontier_search_bench/UPSTREAM.md) for +the ownership and isolation boundaries. The benchmark implementation consumes +unified JSON answer files; inside FrontierAgent, the shared subprocess runner +collects and exports those answers as described below. No dataset download is needed. The 41 queries and all 41 official scorers are -bundled with the subtree. +bundled with FrontierAgent. ## Evaluation isolation requirement @@ -46,7 +46,7 @@ and prints a warning into the run's own output; it is not evidence of isolation. ## Credentials -The wrapper maps `JUDGE_API_KEY` and `JUDGE_BASE_URL` onto the upstream scorer's +The wrapper maps `JUDGE_API_KEY` and `JUDGE_BASE_URL` onto the scorer's `OPENROUTER_API_KEY` and `OPENROUTER_BASE_URL`, always as a pair from one provider — set `OPENROUTER_*` directly and the `JUDGE_*` values are left alone. @@ -81,7 +81,7 @@ interrupted collection continues where it stopped. ## 2. Export -Export the run into the upstream unified JSON contract. For a multi-run +Export the run into the benchmark's unified JSON contract. For a multi-run evaluation, export each `run_` separately as one model/run input. ```bash @@ -98,8 +98,9 @@ of 41 would report a headline score for the 12 easiest. Finish the run, or pass ## 3. Score Run all 41 official scorers. The wrapper reuses `JUDGE_API_KEY` and -`JUDGE_BASE_URL`, executes a temporary copy of the upstream tree, and preserves -aggregate plus per-query artifacts under the requested result directory. +`JUDGE_BASE_URL`, executes a temporary copy of the benchmark implementation, +and preserves aggregate plus per-query artifacts under the requested result +directory. ```bash uv run python -m benchmarks.public.runner.score_frontier_search \ @@ -128,9 +129,9 @@ an explicit step. ## Scoring internals -Upstream's own documentation is the reference for what the scorers do: +The bundled scorer documentation is the reference for what the scorers do: [`eval/verifiable/README.md`](../benchmarks/frontier_search_bench/eval/verifiable/README.md) covers the CLI, the three-stage pipeline, normalization, and the meaning of coverage. The unified JSON answer contract it consumes — and which -`export_frontier_search` produces — is documented in the subtree's +`export_frontier_search` produces — is documented in the benchmark's [`README.md`](../benchmarks/frontier_search_bench/README.md). diff --git a/docs/framework.md b/docs/framework.md index 45fcd59..7179cb5 100644 --- a/docs/framework.md +++ b/docs/framework.md @@ -10,7 +10,7 @@ plugins, and benchmark evaluation. The framework layer has no dependency on frontier_agent/ generic loop, scheduling, registries, AgentBus, observers plugins/tools/ tool implementations and sandbox policy workflows/ pipeline specs, profiles, prompts, workflow-owned observers -benchmarks/ public benchmark harness plus standalone FrontierSearchBench +benchmarks/ public harness plus bundled FrontierSearchBench/FrontierChallenge ``` ## Runtime flow From 223cfcb925c9d87ded8eed691dca94a880c5cf9e Mon Sep 17 00:00:00 2001 From: Yue Deng Date: Tue, 25 Aug 2026 17:46:33 +0800 Subject: [PATCH 2/2] docs: correct FrontierSearchBench env location --- benchmarks/frontier_search_bench/eval/verifiable/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/benchmarks/frontier_search_bench/eval/verifiable/README.md b/benchmarks/frontier_search_bench/eval/verifiable/README.md index f931ef1..4dab285 100644 --- a/benchmarks/frontier_search_bench/eval/verifiable/README.md +++ b/benchmarks/frontier_search_bench/eval/verifiable/README.md @@ -89,9 +89,9 @@ cp .env.example .env # edit .env and fill in OPENROUTER_API_KEY=sk-... ``` -`.env` may live in `eval/verifiable/`, its parent directory, or the -FrontierSearchBench directory root — all three locations are loaded -automatically. +Place `.env` at `eval/verifiable/.env`, the location consistently supported by +the batch runner and all query scorers. Alternatively, export +`OPENROUTER_API_KEY` and `OPENROUTER_BASE_URL` in the environment. The code reads two environment variables: `OPENROUTER_API_KEY` (required) and `OPENROUTER_BASE_URL` (optional, default `https://openrouter.ai/api/v1`). Models are called through the OpenAI SDK.