Quarq Benchmarks

Long-memory benchmark runners for Quarq Agent.

This repository is focused on evaluating the open-source Quarq Agent against LongMemEval-S. The benchmark runner starts from a clean agent memory state for each question, feeds the relevant conversation history through the agent API, asks the final benchmark question, judges the answer, and writes resumable local reports.

Repository links:

Agent: https://github.com/quarqlabs/agent-oss
Benchmarks: https://github.com/quarqlabs/benchmarks

The benchmark runner does not import the agent code directly. It expects an agent-oss FastAPI server to be running and talks to it over HTTP.

Cost warning: the full 500-question LongMemEval-S run is expensive. Our observed full run cost with the current agent model mix was about $2,500, or about $5 per question. Do a 1-question or small-sample run first before spending on the full dataset.

What This Repo Measures

Long-memory agents are tested on whether they can absorb conversation history, store useful memories, retrieve the right evidence later, and answer questions without guessing from unrelated context.

This benchmark focuses on the same failure modes described in the Quarq Agent README:

retrieving the wrong memory
retrieving the right memory but attaching it to the wrong entity
confusing storage time with event time
using unrelated nearby numbers in calculations
missing preference constraints, procedural instructions, or temporal anchors

The current target dataset is LongMemEval-S:

eval_datasets/longmemeval_s_cleaned.json

Each example contains haystack conversations, haystack dates, a final question, an expected answer, and a question type such as multi-session, temporal-reasoning, or knowledge-update.

Evaluation Architecture

The benchmark is intentionally separated from the agent runtime.

benchmarks/run_dataset_evals.py
    |
    | HTTP
    v
agent-oss/main.py FastAPI server
    |
    v
agent-oss/agent_connector.py
    |
    v
agent-oss/agent.py LangGraph agent
    |
    v
local FAISS + JSON memory

The benchmark runner performs orchestration only. The agent repository owns retrieval, learning, memory storage, generation, benchmark mode, and the wipe operation.

This matters because benchmark users can clone the public agent repository, start the agent API, then clone this benchmark repository and run evaluations without relying on local Python imports between repos.

Requirements

Python 3.11 or higher
An OpenAI API key
A working local clone of https://github.com/quarqlabs/agent-oss
A running agent-oss FastAPI server
This benchmark repository: https://github.com/quarqlabs/benchmarks

The benchmark and the agent can use separate Python virtual environments. That is the recommended setup because the benchmark should behave like an external client of the agent API.

Token And Cost Planning

Benchmark cost depends on the model configuration inside the running agent-oss server, not only this runner. Do a 1-question or small-sample run before running all 500 questions.

Observed cost with the current agent model mix:

Run size	Approximate cost
1 average question	about `$5`
10 questions	about `$50`
100 questions	about `$500`
Full 500-question LongMemEval-S run	about `$2,500`

This observed cost is much higher than the direct prompt-token lower bound because each chunk can trigger retrieval planning, generation, multiple gpt-4.1 memory-learning calls, embeddings, final question answering, and gpt-5 judging.

The current local longmemeval_s_cleaned.json file contains:

Item	Value
Questions	500
Total chunks at `chunk_size=8`	41,813
Average chunks per question	83.6
Median chunks per question	83
P90 chunks per question	90
Average characters per chunk ingestion prompt	5,954
Estimated tokens per chunk ingestion prompt	1,489

Token counts above use the common planning approximation:

estimated_tokens = characters / 4

Use them as a budget estimate, not an exact bill. Exact tokenizer counts vary by model and may differ from this character-based estimate.

Per-Question Estimate

If one question has about 80 chunks:

80 chunks * 1,489 tokens/chunk = about 119,000 input tokens

Using the measured average of 83.6 chunks per question:

83.6 chunks * 1,489 tokens/chunk = about 124,500 input tokens

This is only the direct chunk-ingestion prompt traffic sent by the benchmark runner. It does not include the agent's internal retrieval-planning prompts, memory-learning prompts, embeddings, final answer generation, or judge prompt.

Full LongMemEval-S Estimate

For the full 500-question dataset:

500 questions * 80 chunks/question * 1,489 tokens/chunk
= about 59.6M input tokens

Using the measured dataset total:

41,813 chunks * 1,489 tokens/chunk
= about 62.3M input tokens

Pricing Formula

OpenAI API prices are quoted per 1M tokens. Use:

cost = (input_tokens / 1,000,000 * input_price_per_1m)
     + (output_tokens / 1,000,000 * output_price_per_1m)

Because agent-oss does retrieval planning, background memory learning, embedding, and final answer generation, the direct chunk traffic is only a lower bound. The full run will be higher.

The current agent and benchmark model mix is:

Component	Model	Input / 1M	Output / 1M	Cached input / 1M	Notes
Retrieval planning	`gpt-4o-mini`	`$0.15`	`$0.60`	`$0.075`	Used by `retrieval_llm`.
Generation	`gpt-4.1`	`$2.00`	`$8.00`	`$0.50`	Used by `gen_llm`.
Memory learning	`gpt-4.1`	`$2.00`	`$8.00`	`$0.50`	Used by `learn_llm`; usually the major cost driver.
Embeddings	`text-embedding-3-large`	`$0.13`	n/a	n/a	Used by `embed_client`.
Benchmark judge	`gpt-5`	`$1.25`	`$10.00`	`$0.125`	Used only by `judge_llm` in this repo.

Check the OpenAI pricing page before a real run, because model prices can change.

Lower-Bound Input Cost

If the measured 62.3M chunk-ingestion tokens were billed once as plain model input, the input-only costs would be:

Billing path	Price / 1M input tokens	62.3M input-only cost
`gpt-4o-mini` retrieval-planning scale	`$0.15`	about `$9.34`
`gpt-4.1` generation/learning scale	`$2.00`	about `$124.51`
`text-embedding-3-large` embedding scale	`$0.13`	about `$8.09`

For one average question, the direct 124,500-token chunk-ingestion lower bound would be only about $0.25 if billed once as gpt-4.1 input:

124,500 / 1,000,000 * $2.00 = about $0.25

That is why this table is not the final bill. It is a lower-bound planning number. In a real agent run, each chunk can trigger multiple model calls:

gpt-4o-mini for search planning
gpt-4.1 for the ingestion acknowledgement/generation path
gpt-4.1 for memory extraction and consolidation
text-embedding-3-large for retrieval and memory writes

So a practical budget should multiply the gpt-4.1 input estimate by the number of times the chunk content is re-read by learning prompts, then add output tokens and embedding calls.

Cached-input pricing can reduce cost only when the API actually bills repeated prefixes as cached input. Do not assume the whole benchmark gets cached-input pricing; use the returned usage metrics after a run to verify cache hits.

Judge Cost

The judge runs once per completed question with gpt-5. The current judge prompt is about 1,736 estimated input tokens per question, including the rubric, question, expected answer, and a moderate-length agent answer.

For 500 questions:

500 * 1,736 = about 868,000 judge input tokens

At $1.25 per 1M input tokens, judge input is about $1.09. Visible judge output is only YES or NO, but reasoning_effort="medium" may create billable reasoning/output tokens, so leave extra budget for judge output.

Output-token sensitivity for the models used here:

Output tokens	`gpt-4o-mini`	`gpt-4.1`	`gpt-5` judge
1M output tokens	`$0.60`	`$8.00`	`$10.00`
5M output tokens	`$3.00`	`$40.00`	`$50.00`
10M output tokens	`$6.00`	`$80.00`	`$100.00`

The runner stores the agent API metrics object in reports/longmemeval_results.json when the API returns it. Use those recorded metrics for post-run accounting whenever possible.

Agent Setup

Clone and configure the agent first.

git clone https://github.com/quarqlabs/agent-oss.git
cd agent-oss

Follow the setup instructions in the agent repository README. In short, create a Python environment, install dependencies, and provide the required environment variables.

Example local setup:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env

At minimum, the FastAPI worker needs:

OPENAI_API_KEY=your_openai_api_key
USER_ID=eval_user
AGENT_ID=longmemeval_eval
LOCAL_MEMORY_ROOT=local_memory

AGENT_ID selects the local memory namespace under the agent repository. Reusing the same AGENT_ID reuses the same FAISS/JSON memory folder. For benchmark runs, use a dedicated AGENT_ID so normal development memories are not mixed with eval memories.

Start the agent API:

uvicorn main:app --host 127.0.0.1 --port 8000

The benchmark expects these routes to exist in the running agent:

GET / for health
POST /api/chat for feeding chunks and asking questions
POST /api/memories/wipe for clearing agent memory between questions

You can quickly verify the server:

curl http://127.0.0.1:8000/

Expected shape:

{"status":"ok","user_id":"eval_user"}

Benchmark Setup

Clone this repository separately.

git clone https://github.com/quarqlabs/benchmarks.git
cd benchmarks

Create and activate a Python environment:

python3 -m venv .venv
source .venv/bin/activate

Install benchmark dependencies:

pip install -r requirements.txt

Create a local .env file for the benchmark runner:

OPENAI_API_KEY=your_openai_api_key
AGENT_API_BASE_URL=http://127.0.0.1:8000
AGENT_REQUEST_TIMEOUT=300

OPENAI_API_KEY is used by the benchmark judge. The agent server also needs its own OPENAI_API_KEY in the agent repository environment.

AGENT_API_BASE_URL points the benchmark runner at the running agent API. The default is http://127.0.0.1:8000, so this variable is optional if you use the default port.

AGENT_REQUEST_TIMEOUT is the per-request HTTP timeout in seconds. Long memory ingestion and final answer generation can be slow, so keep this high enough for your model and machine.

Running LongMemEval-S

Start the agent API first:

cd path/to/agent-oss
source .venv/bin/activate
uvicorn main:app --host 127.0.0.1 --port 8000

Then run the benchmark:

cd path/to/benchmarks
source .venv/bin/activate
AGENT_API_BASE_URL=http://127.0.0.1:8000 python3 run_dataset_evals.py

The runner prints progress as it:

connects to the agent API
loads the LongMemEval-S dataset
wipes agent memory before a new question
feeds conversation-history chunks
asks the final benchmark question
judges the answer
writes the result row
clears the checkpoint
wipes memory again before moving on

Results are written to:

reports/longmemeval_results.json

The active checkpoint is written to:

reports/eval_checkpoint.json

How The Runner Works

The main entry point is:

run_dataset_evals.py

The runner is deliberately sequential. It evaluates one question at a time so each question starts from a clean memory state and uses the same single-tenant agent API process.

1. Dataset Loading

The runner reads:

eval_datasets/longmemeval_s_cleaned.json

If the file is missing or invalid, the script attempts to download the cleaned dataset file from the configured dataset URL.

2. Completed Question Skip

Before running, the script reads existing results from:

reports/longmemeval_results.json

Any question_id already present in that file is skipped. This makes interrupted runs resumable without deleting successful results.

3. Memory Isolation

When the runner starts a brand-new question, it calls:

POST /api/memories/wipe

The agent API clears semantic memory, episodic memory, and procedural rules for the configured AGENT_ID.

The runner also calls the wipe route after a completed question. This keeps the agent blank before the next question begins.

4. Chunking Conversation History

Each LongMemEval example contains multiple haystack sessions. The runner chunks each session into groups of eight messages.

Each chunk preserves:

the chunk text
the haystack date
the haystack session ID
the session index

The prompt sent to the agent for each chunk is:

Review and remember this conversation history:

<chunk text>

The agent recognizes this ingestion prefix and treats it as memory-learning input instead of a normal user question.

5. Feeding Memory

Each chunk is sent to:

POST /api/chat

with:

{
  "channel_type": "benchmark",
  "skip_learning": false,
  "current_date": "<haystack date>"
}

skip_learning=false lets the agent learn from the chunk. current_date provides the simulated conversation date so relative dates can be resolved against the dataset timeline rather than the machine clock.

After every successful chunk, the runner saves:

{"question_id":"<id>","last_chunk_index":<index>}

to reports/eval_checkpoint.json.

6. Asking The Final Question

After all chunks are fed, the runner asks the final benchmark question through the same /api/chat route.

The final question request uses:

{
  "channel_type": "benchmark",
  "skip_learning": true,
  "current_date": "<question date>"
}

skip_learning=true disables learning for the final benchmark question. The agent should answer from the memories it already learned from the haystack, not learn the expected question as a new memory.

The agent's benchmark retrieval path waits for pending background learning tasks before answering final questions. This prevents the final question from racing ahead while recent chunks are still being saved.

The runner keeps the dataset question_type only in saved result rows for reporting. It is not sent to the agent API.

7. Saving Retrieved Context

The agent API returns:

{
  "response": "...",
  "metrics": {},
  "contexts": {
    "semantic": "...",
    "episodic": "...",
    "procedural": "..."
  }
}

The runner stores the returned context alongside the answer. This makes failures easier to inspect because each result row includes the memories the agent used.

API Contract

The benchmark runner expects the agent API to expose the following contract.

Health

GET /

Response:

{
  "status": "ok",
  "user_id": "eval_user"
}

Chat

POST /api/chat

Request:

{
  "prompt": "What should the agent process?",
  "channel_type": "benchmark",
  "skip_learning": false,
  "current_date": "2024-01-15"
}

Fields:

Field	Required	Description
`prompt`	yes	Chunk ingestion prompt or final benchmark question.
`channel_type`	no	Use `benchmark` for benchmark runs. Defaults to `web` in the API.
`skip_learning`	no	`false` for haystack chunks, `true` for final questions.
`current_date`	no	Dataset date used as the simulated current date.

Response:

{
  "response": "Agent answer",
  "metrics": {},
  "contexts": {
    "semantic": "retrieved semantic memories",
    "episodic": "retrieved episodic memories",
    "procedural": "retrieved procedural rules"
  }
}

Wipe Memories

POST /api/memories/wipe

Response:

{
  "status": "ok",
  "user_id": "eval_user"
}

The wipe route is benchmark-critical. Without it, one question can leak memory into the next question and inflate or corrupt results.

Dataset

The benchmark uses LongMemEval-S cleaned data:

eval_datasets/longmemeval_s_cleaned.json

The expected item shape is:

{
  "question_id": "...",
  "question": "...",
  "answer": "...",
  "question_type": "...",
  "question_date": "...",
  "haystack_sessions": [],
  "haystack_dates": [],
  "haystack_session_ids": []
}

Question types currently reported by the local metrics include:

knowledge-update
multi-session
single-session-assistant
single-session-preference
single-session-user
temporal-reasoning

These categories matter because long-memory failure modes differ by type. Temporal questions need date grounding, preference questions need constraint recall, and multi-session questions often require joining evidence across separate conversations.

Reports And Checkpoints

Results

The result file is:

reports/longmemeval_results.json

Each result row contains:

{
  "question_id": "...",
  "question_type": "...",
  "question": "...",
  "expected_answer": "...",
  "agent_answer": "...",
  "result": "YES",
  "metrics": {},
  "retrieved_context": {
    "semantic": "...",
    "episodic": "...",
    "procedural": "..."
  }
}

result is the binary judge verdict. YES means the answer passed according to the benchmark judge rubric. NO means the answer missed the core factual answer or violated a core constraint.

Checkpoint

The checkpoint file is:

reports/eval_checkpoint.json

Shape:

{
  "question_id": "example_question_id",
  "last_chunk_index": 12
}

If a run is interrupted while feeding chunks, rerunning the script resumes from the next chunk for the same question. Once a question finishes and is judged, the checkpoint is reset to:

{"question_id":null,"last_chunk_index":-1}

Resuming Runs

The runner uses two resume mechanisms:

completed question IDs from reports/longmemeval_results.json
current chunk progress from reports/eval_checkpoint.json

This lets long runs survive process stops, API failures, rate limits, or machine restarts.

Judging

The benchmark uses an LLM binary judge in run_dataset_evals.py.

The judge compares:

the benchmark question
the expected answer
the agent answer

It returns only:

YES

or:

NO

The rubric is intentionally semantic rather than exact-string based. It accepts answers that:

contain or clearly mean the expected answer
provide the expected answer in a full sentence
use tables, bullets, or structured formatting
include correct extra context
satisfy a preference or advice request without repeating every detail
identify the same missing variable in partial-data questions
express negative evidence as "not mentioned" or "no indication"

It rejects answers that:

miss the core factual information
contradict the expected answer
violate a hard preference or constraint
guess when the required evidence is absent
use the wrong numeric scalar for an exact target

The default judge model in the current script is:

gpt-5

with:

reasoning_effort=medium
temperature=0

Metrics

The current local LongMemEval-S metrics from the agent README are computed from reports/longmemeval_results.json joined with eval_datasets/longmemeval_s_cleaned.json by question_id.

These are local progress metrics while Quarq Agent is actively being improved. Treat checked-in report files as local progress snapshots, not final published benchmark numbers.

Question type	Correct	Incorrect	Total	Accuracy
Overall	491	9	500	98.20%
knowledge-update	77	1	78	98.72%
multi-session	129	4	133	96.99%
single-session-assistant	56	0	56	100.00%
single-session-preference	30	0	30	100.00%
single-session-user	70	0	70	100.00%
temporal-reasoning	129	4	133	96.99%

Accuracy is calculated as:

correct / total

The result set uses binary judge labels:

YES counts as correct
NO counts as incorrect

Environment Variables

Benchmark Variables

Variable	Required	Default	Description
`OPENAI_API_KEY`	yes	none	Used by the benchmark judge model.
`AGENT_API_BASE_URL`	no	`http://127.0.0.1:8000`	Base URL for the running `agent-oss` FastAPI server.
`AGENT_REQUEST_TIMEOUT`	no	`300`	HTTP timeout in seconds for agent API requests.

Agent Variables

The agent repository has its own environment. The benchmark requires the agent API to be running, so the agent needs its normal configuration.

Variable	Required	Description
`OPENAI_API_KEY`	yes	Used by Quarq Agent for generation, retrieval planning, learning, and embeddings.
`USER_ID`	yes for API	Required by `agent-oss/main.py`.
`AGENT_ID`	recommended	Selects the local memory namespace. Use a benchmark-specific value.
`LOCAL_MEMORY_ROOT`	no	Root folder for local FAISS/JSON memory stores.

Troubleshooting

The benchmark cannot connect to the agent API

Check that the agent server is running:

curl http://127.0.0.1:8000/

If your server uses a different port:

AGENT_API_BASE_URL=http://127.0.0.1:9000 python3 run_dataset_evals.py

The agent API says `USER_ID environment variable is required`

Set USER_ID in the agent repository environment before starting FastAPI:

USER_ID=eval_user uvicorn main:app --host 127.0.0.1 --port 8000

Results look contaminated across questions

Use a dedicated benchmark AGENT_ID, then restart the agent API:

AGENT_ID=longmemeval_eval USER_ID=eval_user uvicorn main:app --host 127.0.0.1 --port 8000

The runner calls /api/memories/wipe before new questions and after completed questions. If the wipe route is missing or failing, memory isolation is broken.

Requests time out

Increase the benchmark timeout:

AGENT_REQUEST_TIMEOUT=600 python3 run_dataset_evals.py

Also check the agent server logs. Long chunk ingestion may still be running even when the benchmark side times out.

A run stopped in the middle of a question

Rerun the same command. The runner will read:

reports/eval_checkpoint.json

and continue from the next chunk for the same question.

You want to rerun everything from scratch

Move or delete:

reports/longmemeval_results.json
reports/eval_checkpoint.json

Then run the benchmark again. The agent memory will still be wiped through the API at question boundaries.

Repository Map

README.md                              Benchmark setup and operating guide
requirements.txt                       Python dependencies for the benchmark runner
run_dataset_evals.py                   Sequential LongMemEval-S API runner
eval_datasets/longmemeval_s_cleaned.json
                                       Cleaned LongMemEval-S dataset
reports/eval_checkpoint.json           Current in-progress question/chunk checkpoint
reports/longmemeval_results.json       Main benchmark result file

Notes For Contributors

Keep the benchmark runner and the agent runtime separated. The benchmark should continue to treat the agent as an HTTP service, not as an importable local module.

When changing the runner, preserve:

per-question memory wipe
chunk-level checkpointing
current_date propagation
final-question skip_learning=true
returned retrieved-context storage
binary judge output normalization

Those details are part of what makes the results reproducible and inspectable.

Name		Name	Last commit message	Last commit date
reports		reports
.env.example		.env.example
.gitignore		.gitignore
README.md		README.md
requirements.txt		requirements.txt
run_dataset_evals.py		run_dataset_evals.py

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

Quarq Benchmarks

Contents

What This Repo Measures

Evaluation Architecture

Requirements

Token And Cost Planning

Per-Question Estimate

Full LongMemEval-S Estimate

Pricing Formula

Lower-Bound Input Cost

Judge Cost

Agent Setup

Benchmark Setup

Running LongMemEval-S

How The Runner Works

1. Dataset Loading

2. Completed Question Skip

3. Memory Isolation

4. Chunking Conversation History

5. Feeding Memory

6. Asking The Final Question

7. Saving Retrieved Context

API Contract

Health

Chat

Wipe Memories

Dataset

Reports And Checkpoints

Results

Checkpoint

Resuming Runs

Judging

Metrics

Environment Variables

Benchmark Variables

Agent Variables

Troubleshooting

The benchmark cannot connect to the agent API

The agent API says USER_ID environment variable is required

Results look contaminated across questions

Requests time out

A run stopped in the middle of a question

You want to rerun everything from scratch

Repository Map

Notes For Contributors

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

The agent API says `USER_ID environment variable is required`