diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
index fb8dcce8..62a3f09a 100644
--- a/.pre-commit-config.yaml
+++ b/.pre-commit-config.yaml
@@ -59,6 +59,9 @@ repos:
- "**/build/**"
- "-ignore"
- "**/vendor/**"
+ # Separately licensed Apache-2.0 project
+ - "-ignore"
+ - "labs/AgentStream/**"
# Generated protobuf / gRPC stubs
- "-ignore"
- "**/*.pb.go"
diff --git a/CHANGELOG.md b/CHANGELOG.md
index c282d64e..a24c533b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -24,6 +24,9 @@ Guidelines for editors:
### Added
+- **AgentStream:** add an experimental streaming evaluation framework for
+ self-evolving LLM agents under `labs/AgentStream` (#72).
+
### Changed
### Deprecated
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 9ce381a3..ed234c3e 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -1,8 +1,10 @@
# Contributing to Sico
-Thanks for your interest in contributing! This project is MIT-licensed: by
-submitting a pull request you agree that your contribution will be distributed
-under the same terms.
+Thanks for your interest in contributing! Sico is MIT-licensed, except for
+[`labs/AgentStream`](labs/AgentStream), which is separately licensed under the
+Apache License 2.0. By submitting a pull request, you agree that your
+contribution will be distributed under the license applicable to the files you
+modify.
## Start here
@@ -53,9 +55,12 @@ service-specific commands, and troubleshooting notes.
## License headers
-Every new source file (Go, Python, TypeScript, JavaScript, proto, shell, YAML,
-Dockerfile, ...) must carry the MIT header. The pre-commit hook adds it
-automatically. Generated files are intentionally excluded; see the
+Every new source file outside `labs/AgentStream` (Go, Python, TypeScript,
+JavaScript, proto, shell, YAML, Dockerfile, ...) must carry the MIT header. The
+pre-commit hook adds it automatically. Files under `labs/AgentStream` instead
+use the Apache-2.0 SPDX header described in the
+[AgentStream contribution guide](labs/AgentStream/exgentic/CONTRIBUTING.md).
+Generated files are intentionally excluded; see the
[Development guide](docs/development.md#license-headers) and
[pre-commit configuration](.pre-commit-config.yaml) for the exact ignore list.
diff --git a/README.md b/README.md
index 17426c92..563b7510 100644
--- a/README.md
+++ b/README.md
@@ -219,6 +219,8 @@ sico/
├── proto/ # Protobuf definitions shared by all services
├── sandbox/ # Sandbox runtimes (Android emulator, ...)
├── examples/ # Runnable workflow examples (auth, LLM Hub, conversation, sandbox, ...)
+├── labs/
+│ └── AgentStream/ # Experimental streaming evaluation for self-evolving LLM agents
├── deploy/
│ ├── docker/ # docker-compose stack
│ └── kind/ # Kind + Helm setup
@@ -252,6 +254,8 @@ Contributions of all kinds are welcome: bug reports, feature ideas, documentatio
Sico is licensed under the [MIT License](LICENSE).
+The contents of [AgentStream](labs/AgentStream) are licensed separately under the [Apache License 2.0](labs/AgentStream/LICENSE).
+
## Trademarks
This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow [Microsoft’s Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks). Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party’s policies.
diff --git a/labs/AgentStream/LICENSE b/labs/AgentStream/LICENSE
new file mode 100644
index 00000000..261eeb9e
--- /dev/null
+++ b/labs/AgentStream/LICENSE
@@ -0,0 +1,201 @@
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/labs/AgentStream/README.md b/labs/AgentStream/README.md
index 58fb6d24..304f324d 100644
--- a/labs/AgentStream/README.md
+++ b/labs/AgentStream/README.md
@@ -24,9 +24,14 @@
dapenghu@microsoft.com
+
+
+
+
## 🚀 News
+* **[2026/08]** Code is released!
* **[2026/07]** Code is under preparation. Stay tuned!
## 📖 Overview
@@ -38,3 +43,101 @@ Over these scenarios, we combinatorially evaluate five representative self-evolv
Our results show that self-evolution reliability varies across streaming scenarios, the benefit of self-evolution is gated by model capability and non-monotonic in model strength, and no single method dominates across models and scenarios.
These findings offer concrete guidance for selecting self-evolving methods across models and streaming scenarios.
Overall, we advocate that self-evolving agents should be evaluated under realistic task streams rather than isolated single-task settings.
+
+
+
+
+
+## ⚡️ Getting Started
+
+AgentStream is built on a locally adapted snapshot of the
+[Exgentic](https://github.com/Exgentic/exgentic) framework, bundled under
+[`exgentic`](./exgentic). This copy adds the self-evolving agents and streaming
+experiment runners used by AgentStream. AgentStream-specific changes to the
+snapshot are maintained in Sico, and the bundled package is not published
+independently from this repository. The five self-evolving agents live under
+[`exgentic/src/exgentic/agents`](./exgentic/src/exgentic/agents), and the
+benchmarks are orchestrated through `exgentic`'s installation and runner
+infrastructure.
+
+### 1. Requirements
+
+- Python `>= 3.11`
+- [`uv`](https://github.com/astral-sh/uv)
+- Docker (optional)
+
+### 2. Install the local exgentic (agent side)
+
+Clone the repo and create an editable environment from the bundled `exgentic`:
+
+```bash
+git clone https://github.com/microsoft/Sico.git
+cd Sico/labs/AgentStream/exgentic
+
+# Install the local ./src/exgentic in editable mode into .venv/
+uv sync
+
+# Activate the environment
+source .venv/bin/activate
+```
+
+Verify that the self-evolving agents are visible from the local install:
+
+```bash
+uv run exgentic list agents
+```
+
+### 3. Install benchmarks (benchmark side)
+
+Each benchmark is installed into isolated **`venv`** environment:
+
+```bash
+cd Sico/labs/AgentStream/exgentic
+
+
+uv run exgentic install --benchmark tau2
+uv run exgentic install --benchmark bfcl
+uv run exgentic install --benchmark hle
+uv run exgentic install --benchmark appworld
+uv run exgentic install --benchmark swebench
+uv run exgentic install --benchmark browsecompplus
+```
+
+### 4. API credentials
+
+The runners call LLMs through [LiteLLM](https://docs.litellm.ai/). Set the credentials for your provider in the [`exgentic/scripts//run_experiment.sh`](./exgentic/scripts/ace/run_experiment.sh):
+
+```bash
+export OPENAI_API_KEY="..."
+export OPENAI_API_BASE="..."
+```
+
+### 5. Run the streaming experiments
+
+Each method has its own runner under [`exgentic/scripts/`](./exgentic/scripts). The shell script selects the streaming scenario via `MODE` (`isolated` | `sequential` | `interleaved`), the model, the seed, and the benchmark stream:
+
+```bash
+cd Sico/labs/AgentStream/exgentic/scripts/ace
+
+bash run_experiment.sh
+```
+
+
+## 🙏 Acknowledgement
+This work is based on [Exgentic](https://github.com/Exgentic/exgentic). We sincerely thank the authors and contributors of these excellent open-source projects.
+
+## 📚 Citation
+If you find our work helpful, please consider citing:
+
+```bibtex
+@article{yan2026agentstream,
+ title={AgentStream: How Well Do Self-Evolving LLM Agents Perform Under Streaming Tasks?},
+ author={Yan, Dong and Liang, Jian and Hu, Dapeng and He, Ran and Yuan, Nicholas Jing and Zhang, Qi and Tan, Tieniu},
+ journal={arXiv preprint arXiv:2608.00155},
+ year={2026}
+}
+```
+
+## 📄 License
+
+The contents of this AgentStream directory are licensed separately under the [Apache License 2.0](./LICENSE).
diff --git a/labs/AgentStream/exgentic/.dockerignore b/labs/AgentStream/exgentic/.dockerignore
new file mode 100644
index 00000000..79975c4c
--- /dev/null
+++ b/labs/AgentStream/exgentic/.dockerignore
@@ -0,0 +1,19 @@
+.venv/
+.git/
+outputs/
+__pycache__/
+*.pyc
+.mypy_cache/
+.pytest_cache/
+.ruff_cache/
+.exgentic/
+tests/
+
+# Large benchmark data/assets — not needed in the base Docker image.
+# Benchmarks that need these should use setup_script or volumes instead.
+src/exgentic/benchmarks/browsecompplus/assets/
+# Tau2: exclude large pre-computed results and figures but keep domain
+# data files (tasks.json, policy.md, db.json) which the session needs.
+src/exgentic/benchmarks/tau2/installation/tau2-bench/data/tau2/results/
+src/exgentic/benchmarks/tau2/installation/tau2-bench/figs/
+src/exgentic/benchmarks/tau2/installation/tau2-bench/tests/
diff --git a/labs/AgentStream/exgentic/.gitattributes b/labs/AgentStream/exgentic/.gitattributes
new file mode 100644
index 00000000..6313b56c
--- /dev/null
+++ b/labs/AgentStream/exgentic/.gitattributes
@@ -0,0 +1 @@
+* text=auto eol=lf
diff --git a/labs/AgentStream/exgentic/.gitignore b/labs/AgentStream/exgentic/.gitignore
new file mode 100644
index 00000000..2609c5fc
--- /dev/null
+++ b/labs/AgentStream/exgentic/.gitignore
@@ -0,0 +1,121 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+!src/exgentic/interfaces/lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+src/exgentic/_version.py
+
+# Exgentic local data (venvs, caches, installations)
+.exgentic/
+.exgentic_installations/
+
+# Virtual environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# AI/CLI agent metadata
+.bob/
+.claude/
+.cursor/
+.aider*
+.copilot/
+.continue/
+
+# VS Code settings
+.vscode/
+.history/
+*.code-workspace
+
+# PyInstaller
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+.jupyter/
+profile_default/
+
+# PyCharm
+.idea/
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pytype
+.pytype/
+
+# Cython debug symbols
+cython_debug/
+outputs/
+# OS files
+.DS_Store
+Thumbs.db
+data/
+src/exgentic/benchmarks/appworld/data/
+src/exgentic/benchmarks/appworld/experiments/
+src/exgentic/benchmarks/tau2/installation/
+
+# Benchmark caches
+tau2_disk_cache/
+
+old_outputs2/
+outputs
+*outputs*
+.litellm_cache
+trace.jsonl
+
+t.*
+node_modules
+package-lock.json
+package.json
+src/exgentic/benchmarks/browsecompplus/assets
+exgentic_session_cache/
+scripts/litellm/docker_vm_data/Ubuntu.qcow2.zip
diff --git a/labs/AgentStream/exgentic/.pre-commit-config.yaml b/labs/AgentStream/exgentic/.pre-commit-config.yaml
new file mode 100644
index 00000000..553d4703
--- /dev/null
+++ b/labs/AgentStream/exgentic/.pre-commit-config.yaml
@@ -0,0 +1,78 @@
+# See https://pre-commit.com for more information
+# See https://pre-commit.com/hooks.html for more hooks
+# These component-specific checks are intentionally run manually; see DEVELOPMENT.md.
+# Paths are evaluated from the Sico repository root.
+files: ^labs/AgentStream/exgentic/
+exclude: ^labs/AgentStream/exgentic/\.venv/
+
+repos:
+ - repo: https://github.com/astral-sh/ruff-pre-commit
+ # Ruff version.
+ rev: v0.1.6
+ hooks:
+ - id: ruff
+ # Keep the bundled snapshot unchanged while checking syntax and
+ # undefined names. The explicit config avoids inheriting Sico's root
+ # Ruff settings when this hook is run from the monorepo root.
+ args:
+ - --config
+ - labs/AgentStream/exgentic/pyproject.toml
+ - --select
+ - E9,F63,F7,F82
+
+ - repo: https://github.com/Yelp/detect-secrets
+ rev: v1.5.0
+ hooks:
+ - id: detect-secrets
+ args: [--baseline, labs/AgentStream/exgentic/misc/utils/.secrets.baseline]
+ exclude: ^labs/AgentStream/exgentic/misc/utils/\.secrets\.baseline$
+
+ - repo: https://github.com/astral-sh/uv-pre-commit
+ rev: 0.7.12
+ hooks:
+ - id: uv-lock
+ args: [--locked, --project, labs/AgentStream/exgentic]
+ files: ^labs/AgentStream/exgentic/(uv\.lock|pyproject\.toml|uv\.toml)$
+
+ - repo: https://github.com/codespell-project/codespell
+ rev: v2.2.6
+ hooks:
+ - id: codespell
+ args: [--toml, labs/AgentStream/exgentic/pyproject.toml]
+ exclude: ^labs/AgentStream/exgentic/(tests/benchmarks/recordings/|uv\.lock$)
+ additional_dependencies:
+ - tomli
+
+ - repo: local
+ hooks:
+ - id: enforce-spdx-header
+ name: Enforce SPDX Header
+ entry: python labs/AgentStream/exgentic/misc/utils/enforce_spdx_header.py
+ language: python
+ files: ^labs/AgentStream/exgentic/.*\.py$
+ types: [python]
+ - id: enforce-relative-imports
+ name: Enforce Relative Imports
+ entry: python labs/AgentStream/exgentic/misc/utils/enforce_relative_imports.py
+ language: python
+ # Adjust the files pattern to match your needs
+ files: ^labs/AgentStream/exgentic/src/.*\.py$
+ # Optional: Specify types or exclude files
+ types: [python]
+ - id: enforce-dependency-caps
+ name: Enforce Dependency Version Caps
+ entry: python labs/AgentStream/exgentic/misc/utils/enforce_dependency_caps.py
+ language: python
+ files: ^labs/AgentStream/exgentic/pyproject\.toml$
+ pass_filenames: false
+
+ - repo: local
+ hooks:
+ - id: enforce-library-imports
+ name: Enforce Library Imports
+ entry: python labs/AgentStream/exgentic/misc/utils/enforce_library_imports.py
+ language: python
+ # Adjust the files pattern to match your needs
+ exclude: ^labs/AgentStream/exgentic/(src/.*\.py|misc/utils/enforce_(library_imports|relative_imports)\.py)$
+ # Optional: Specify types or exclude files
+ types: [python]
diff --git a/labs/AgentStream/exgentic/CONTRIBUTING.md b/labs/AgentStream/exgentic/CONTRIBUTING.md
new file mode 100644
index 00000000..d3939f57
--- /dev/null
+++ b/labs/AgentStream/exgentic/CONTRIBUTING.md
@@ -0,0 +1,32 @@
+# Contributing to the bundled Exgentic component
+
+This directory contains a locally adapted snapshot of
+[Exgentic](https://github.com/Exgentic/exgentic) used by AgentStream. Changes
+specific to this bundled copy are contributed through the
+[Sico repository](https://github.com/microsoft/Sico), not through the upstream
+Exgentic repository.
+
+## Contribution process
+
+Follow Sico's root [contribution guide](../../../CONTRIBUTING.md) for the fork,
+branch, commit, pull-request, review, code-of-conduct, and security-reporting
+processes. For local setup and component-specific checks, see
+[DEVELOPMENT.md](./DEVELOPMENT.md).
+
+If a change applies to the original Exgentic project rather than this
+AgentStream-specific copy, contribute it to the
+[upstream repository](https://github.com/Exgentic/exgentic) separately.
+
+## License
+
+The contents of `labs/AgentStream`, including this bundled component, are
+licensed under the Apache License 2.0; see [LICENSE](./LICENSE) and the
+[AgentStream license](../LICENSE). This is an exception to Sico's root MIT
+license.
+
+New source files in this directory must include the following SPDX identifier
+using the appropriate comment syntax:
+
+```text
+SPDX-License-Identifier: Apache-2.0
+```
diff --git a/labs/AgentStream/exgentic/DEVELOPMENT.md b/labs/AgentStream/exgentic/DEVELOPMENT.md
new file mode 100644
index 00000000..3ccebda9
--- /dev/null
+++ b/labs/AgentStream/exgentic/DEVELOPMENT.md
@@ -0,0 +1,130 @@
+# Development Guide
+
+This guide covers setting up exgentic for local development, editing, and debugging.
+
+## Setup
+
+```bash
+git clone https://github.com/microsoft/Sico.git
+cd Sico/labs/AgentStream/exgentic
+uv sync --frozen
+```
+
+## Setup Benchmarks & Agents
+
+Benchmarks and agents declare their dependencies through two mechanisms:
+
+- **`requirements.txt`** — pip packages installed automatically via `uv pip install`
+- **`setup.sh`** — shell script for non-pip setup (apt packages, git clones, data downloads)
+
+Both are auto-discovered next to the benchmark/agent module directory. The `exgentic install` command runs both:
+
+```bash
+# Benchmarks
+uv run exgentic install --benchmark tau2
+uv run exgentic install --benchmark appworld
+uv run exgentic install --benchmark gsm8k
+uv run exgentic install --benchmark hotpotqa
+uv run exgentic install --benchmark swebench
+uv run exgentic install --benchmark browsecompplus
+
+# Agents
+uv run exgentic install --agent litellm_tool_calling
+uv run exgentic install --agent smolagents
+uv run exgentic install --agent openai
+uv run exgentic install --agent claude
+uv run exgentic install --agent codex
+uv run exgentic install --agent gemini
+```
+
+> **Note:** `exgentic setup` still works but is deprecated. Use `install`/`uninstall` instead.
+
+### Isolated Runners (venv / docker)
+
+By default, benchmarks run with the `venv` runner, which creates an isolated `uv` virtual environment per benchmark under `.exgentic//venv/`. This means **no local setup is needed** — dependencies are installed automatically in the venv on first run.
+
+You can also use the `docker` runner for full container isolation:
+
+```bash
+uv run exgentic evaluate --benchmark tau2 --agent tool_calling --subset retail --num-tasks 2 \
+ --model gpt-4o \
+ --set benchmark.runner=docker \
+ --set benchmark.user_simulator_model="gpt-4o"
+```
+
+Both isolated runners follow the same pattern:
+
+1. Install `requirements.txt` and run `setup.sh` in the isolated environment
+2. Start `exgentic serve --cls --kwargs ` inside the venv/container
+3. Communicate over HTTP via the runner transport layer
+
+Setup scripts can check the `EXGENTIC_DOCKER_BUILD` environment variable to distinguish a Docker build from a local setup (e.g., to skip interactive prompts or large downloads that are handled differently in containers).
+
+## API Credentials
+
+```bash
+export OPENAI_API_KEY=...
+# or
+export ANTHROPIC_API_KEY=...
+```
+
+Or create a `.env` file in the project root — Exgentic loads it automatically.
+
+## Running Evaluations
+
+```bash
+uv run exgentic list benchmarks
+uv run exgentic list agents
+
+uv run exgentic evaluate --benchmark tau2 --agent tool_calling --subset retail --num-tasks 2 \
+ --model gpt-4o \
+ --set benchmark.user_simulator_model="gpt-4o"
+```
+
+## Tests
+
+```bash
+# Core tests (no Docker or external services required)
+uv run pytest tests/ --ignore=tests/integrations --ignore=tests/adapters/runners
+
+# Runner/transport tests (includes Docker tests on matching Python version)
+uv run pytest tests/adapters/runners -v -p no:faulthandler
+
+# API-level tests only
+uv run pytest tests/api
+
+# Skip tests requiring external services
+uv run pytest tests/ -k "not litellm and not mcp"
+```
+
+The test suite includes **replay tests** that re-run recorded benchmark sessions without any external dependencies. Recordings are stored under `tests/benchmarks/recordings/` and use `ReplayBenchmark` + `ReplayAgent` to verify the execution loop end-to-end.
+
+## Linting
+
+These checks are not part of Sico's root CI. Contributors modifying the bundled
+Exgentic snapshot should run them manually from the Sico repository root.
+Install [pre-commit](https://pre-commit.com/#install) for your platform, then run:
+
+```bash
+pre-commit run --config labs/AgentStream/exgentic/.pre-commit-config.yaml --all-files --show-diff-on-failure
+```
+
+## OpenTelemetry Tracing
+
+```bash
+uv sync --extra otel
+
+export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
+export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
+export EXGENTIC_OTEL_ENABLED=true
+```
+
+See [OpenTelemetry semantic conventions](./docs/observability/semantic-conventions.md) for details.
+
+## Packaging
+
+The bundled Exgentic component is installed from source for AgentStream and is
+not published independently from Sico. Do not create Exgentic release tags or
+publish this copy to PyPI from this repository.
+
+For benchmark adapter design guidance, see `docs/adding-benchmarks.md`.
diff --git a/labs/AgentStream/exgentic/LICENSE b/labs/AgentStream/exgentic/LICENSE
new file mode 100644
index 00000000..261eeb9e
--- /dev/null
+++ b/labs/AgentStream/exgentic/LICENSE
@@ -0,0 +1,201 @@
+ Apache License
+ Version 2.0, January 2004
+ http://www.apache.org/licenses/
+
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+ 1. Definitions.
+
+ "License" shall mean the terms and conditions for use, reproduction,
+ and distribution as defined by Sections 1 through 9 of this document.
+
+ "Licensor" shall mean the copyright owner or entity authorized by
+ the copyright owner that is granting the License.
+
+ "Legal Entity" shall mean the union of the acting entity and all
+ other entities that control, are controlled by, or are under common
+ control with that entity. For the purposes of this definition,
+ "control" means (i) the power, direct or indirect, to cause the
+ direction or management of such entity, whether by contract or
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
+ outstanding shares, or (iii) beneficial ownership of such entity.
+
+ "You" (or "Your") shall mean an individual or Legal Entity
+ exercising permissions granted by this License.
+
+ "Source" form shall mean the preferred form for making modifications,
+ including but not limited to software source code, documentation
+ source, and configuration files.
+
+ "Object" form shall mean any form resulting from mechanical
+ transformation or translation of a Source form, including but
+ not limited to compiled object code, generated documentation,
+ and conversions to other media types.
+
+ "Work" shall mean the work of authorship, whether in Source or
+ Object form, made available under the License, as indicated by a
+ copyright notice that is included in or attached to the work
+ (an example is provided in the Appendix below).
+
+ "Derivative Works" shall mean any work, whether in Source or Object
+ form, that is based on (or derived from) the Work and for which the
+ editorial revisions, annotations, elaborations, or other modifications
+ represent, as a whole, an original work of authorship. For the purposes
+ of this License, Derivative Works shall not include works that remain
+ separable from, or merely link (or bind by name) to the interfaces of,
+ the Work and Derivative Works thereof.
+
+ "Contribution" shall mean any work of authorship, including
+ the original version of the Work and any modifications or additions
+ to that Work or Derivative Works thereof, that is intentionally
+ submitted to Licensor for inclusion in the Work by the copyright owner
+ or by an individual or Legal Entity authorized to submit on behalf of
+ the copyright owner. For the purposes of this definition, "submitted"
+ means any form of electronic, verbal, or written communication sent
+ to the Licensor or its representatives, including but not limited to
+ communication on electronic mailing lists, source code control systems,
+ and issue tracking systems that are managed by, or on behalf of, the
+ Licensor for the purpose of discussing and improving the Work, but
+ excluding communication that is conspicuously marked or otherwise
+ designated in writing by the copyright owner as "Not a Contribution."
+
+ "Contributor" shall mean Licensor and any individual or Legal Entity
+ on behalf of whom a Contribution has been received by Licensor and
+ subsequently incorporated within the Work.
+
+ 2. Grant of Copyright License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ copyright license to reproduce, prepare Derivative Works of,
+ publicly display, publicly perform, sublicense, and distribute the
+ Work and such Derivative Works in Source or Object form.
+
+ 3. Grant of Patent License. Subject to the terms and conditions of
+ this License, each Contributor hereby grants to You a perpetual,
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+ (except as stated in this section) patent license to make, have made,
+ use, offer to sell, sell, import, and otherwise transfer the Work,
+ where such license applies only to those patent claims licensable
+ by such Contributor that are necessarily infringed by their
+ Contribution(s) alone or by combination of their Contribution(s)
+ with the Work to which such Contribution(s) was submitted. If You
+ institute patent litigation against any entity (including a
+ cross-claim or counterclaim in a lawsuit) alleging that the Work
+ or a Contribution incorporated within the Work constitutes direct
+ or contributory patent infringement, then any patent licenses
+ granted to You under this License for that Work shall terminate
+ as of the date such litigation is filed.
+
+ 4. Redistribution. You may reproduce and distribute copies of the
+ Work or Derivative Works thereof in any medium, with or without
+ modifications, and in Source or Object form, provided that You
+ meet the following conditions:
+
+ (a) You must give any other recipients of the Work or
+ Derivative Works a copy of this License; and
+
+ (b) You must cause any modified files to carry prominent notices
+ stating that You changed the files; and
+
+ (c) You must retain, in the Source form of any Derivative Works
+ that You distribute, all copyright, patent, trademark, and
+ attribution notices from the Source form of the Work,
+ excluding those notices that do not pertain to any part of
+ the Derivative Works; and
+
+ (d) If the Work includes a "NOTICE" text file as part of its
+ distribution, then any Derivative Works that You distribute must
+ include a readable copy of the attribution notices contained
+ within such NOTICE file, excluding those notices that do not
+ pertain to any part of the Derivative Works, in at least one
+ of the following places: within a NOTICE text file distributed
+ as part of the Derivative Works; within the Source form or
+ documentation, if provided along with the Derivative Works; or,
+ within a display generated by the Derivative Works, if and
+ wherever such third-party notices normally appear. The contents
+ of the NOTICE file are for informational purposes only and
+ do not modify the License. You may add Your own attribution
+ notices within Derivative Works that You distribute, alongside
+ or as an addendum to the NOTICE text from the Work, provided
+ that such additional attribution notices cannot be construed
+ as modifying the License.
+
+ You may add Your own copyright statement to Your modifications and
+ may provide additional or different license terms and conditions
+ for use, reproduction, or distribution of Your modifications, or
+ for any such Derivative Works as a whole, provided Your use,
+ reproduction, and distribution of the Work otherwise complies with
+ the conditions stated in this License.
+
+ 5. Submission of Contributions. Unless You explicitly state otherwise,
+ any Contribution intentionally submitted for inclusion in the Work
+ by You to the Licensor shall be under the terms and conditions of
+ this License, without any additional terms or conditions.
+ Notwithstanding the above, nothing herein shall supersede or modify
+ the terms of any separate license agreement you may have executed
+ with Licensor regarding such Contributions.
+
+ 6. Trademarks. This License does not grant permission to use the trade
+ names, trademarks, service marks, or product names of the Licensor,
+ except as required for reasonable and customary use in describing the
+ origin of the Work and reproducing the content of the NOTICE file.
+
+ 7. Disclaimer of Warranty. Unless required by applicable law or
+ agreed to in writing, Licensor provides the Work (and each
+ Contributor provides its Contributions) on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+ implied, including, without limitation, any warranties or conditions
+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+ PARTICULAR PURPOSE. You are solely responsible for determining the
+ appropriateness of using or redistributing the Work and assume any
+ risks associated with Your exercise of permissions under this License.
+
+ 8. Limitation of Liability. In no event and under no legal theory,
+ whether in tort (including negligence), contract, or otherwise,
+ unless required by applicable law (such as deliberate and grossly
+ negligent acts) or agreed to in writing, shall any Contributor be
+ liable to You for damages, including any direct, indirect, special,
+ incidental, or consequential damages of any character arising as a
+ result of this License or out of the use or inability to use the
+ Work (including but not limited to damages for loss of goodwill,
+ work stoppage, computer failure or malfunction, or any and all
+ other commercial damages or losses), even if such Contributor
+ has been advised of the possibility of such damages.
+
+ 9. Accepting Warranty or Additional Liability. While redistributing
+ the Work or Derivative Works thereof, You may choose to offer,
+ and charge a fee for, acceptance of support, warranty, indemnity,
+ or other liability obligations and/or rights consistent with this
+ License. However, in accepting such obligations, You may act only
+ on Your own behalf and on Your sole responsibility, not on behalf
+ of any other Contributor, and only if You agree to indemnify,
+ defend, and hold each Contributor harmless for any liability
+ incurred by, or claims asserted against, such Contributor by reason
+ of your accepting any such warranty or additional liability.
+
+ END OF TERMS AND CONDITIONS
+
+ APPENDIX: How to apply the Apache License to your work.
+
+ To apply the Apache License to your work, attach the following
+ boilerplate notice, with the fields enclosed by brackets "[]"
+ replaced with your own identifying information. (Don't include
+ the brackets!) The text should be enclosed in the appropriate
+ comment syntax for the file format. We also recommend that a
+ file or class name and description of purpose be included on the
+ same "printed page" as the copyright notice for easier
+ identification within third-party archives.
+
+ Copyright [yyyy] [name of copyright owner]
+
+ Licensed under the Apache License, Version 2.0 (the "License");
+ you may not use this file except in compliance with the License.
+ You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
diff --git a/labs/AgentStream/exgentic/README.md b/labs/AgentStream/exgentic/README.md
new file mode 100644
index 00000000..ca235afd
--- /dev/null
+++ b/labs/AgentStream/exgentic/README.md
@@ -0,0 +1,263 @@
+
+
+> [!NOTE]
+> This directory contains a locally adapted snapshot of the upstream
+> [Exgentic](https://github.com/Exgentic/exgentic) project used by
+> [AgentStream](../README.md). AgentStream-specific changes to this copy are
+> maintained in Sico, and it is not published as a separate Exgentic package
+> from this repository. Report issues with this bundled copy in the
+> [Sico issue tracker](https://github.com/microsoft/Sico/issues); report issues
+> with the original project to the upstream repository.
+
+
+ Evaluate any agent on any benchmark in the simplest way possible
+
+
+---
+
+## What is Exgentic?
+
+Exgentic is a universal evaluation framework that enables standardized testing of AI agents across diverse benchmarks and domains. It provides a consistent interface for evaluating any agent on any benchmark, making it easy to compare performance, reproduce results, and ensure your agent works reliably across different tasks and environments.
+
+## Who is it for?
+
+1. **General Audience** - Visit [www.exgentic.ai](https://www.exgentic.ai) to explore the first general agent leaderboard comparing leading agents and frontier models across varied tasks.
+2. **Agent Builders** - Evaluate your agents comprehensively across multiple domains and benchmarks.
+3. **Researchers & Component Developers** - Test agentic components (memory, context compression, planning) across different agents and domains.
+4. **Benchmark Builders** - Evaluate your benchmark across multiple agents to ensure meaningful differentiation.
+
+---
+
+## Quick Start
+
+### Installation
+
+```bash
+git clone https://github.com/microsoft/Sico.git
+cd Sico/labs/AgentStream/exgentic
+uv sync --frozen
+source .venv/bin/activate
+```
+
+The commands below assume that this local environment is active. Alternatively,
+prefix each command with `uv run`.
+
+### API Credentials
+
+```bash
+export OPENAI_API_KEY=...
+# or
+export ANTHROPIC_API_KEY=...
+```
+
+### Run an Evaluation
+
+```bash
+# List available benchmarks and agents
+exgentic list benchmarks
+exgentic list agents
+
+# Evaluate an agent on a benchmark
+exgentic evaluate --benchmark tau2 --agent tool_calling --subset retail --num-tasks 2 \
+ --model gpt-4o \
+ --set benchmark.user_simulator_model="gpt-4o"
+```
+
+Benchmarks are automatically installed on first run — no manual installation needed. You can also install them explicitly:
+
+```bash
+exgentic install --benchmark tau2 # install deps + data (default)
+exgentic install --agent tool_calling
+exgentic install --benchmark tau2 --docker # build Docker image
+exgentic install --benchmark tau2 --local # install into local environment
+exgentic uninstall --benchmark tau2 # remove installed environment
+```
+
+> **Note:** `exgentic setup` still works but is deprecated in favor of `install`/`uninstall`.
+
+For full container isolation, use the Docker runner (`--set benchmark.runner=docker`). You only need Docker installed and running:
+
+```bash
+exgentic evaluate --benchmark tau2 --agent tool_calling --subset retail --num-tasks 2 \
+ --model gpt-4o \
+ --set benchmark.runner=docker \
+ --set benchmark.user_simulator_model="gpt-4o"
+```
+
+### Python API
+
+After syncing the bundled environment above, use exgentic as a library:
+
+```python
+from exgentic import evaluate
+
+results = evaluate(
+ benchmark="tau2",
+ agent="tool_calling",
+ subset="retail",
+ num_tasks=2,
+ model="gpt-4o",
+ benchmark_kwargs={"user_simulator_model": "gpt-4o"},
+)
+```
+
+For more examples, see the [`examples/`](./examples/) directory.
+
+---
+
+## Available Benchmarks
+
+```bash
+exgentic list benchmarks
+```
+
+| Benchmark | Description |
+|-----------|-------------|
+| **tau2** | Simulated customer support tasks across multiple domains (mock, retail, airline, telecom) |
+| **appworld** | Multi-app API environment testing agents' ability to interact with application interfaces |
+| **browsecompplus** | Web search and browsing benchmark for information retrieval and navigation |
+| **swebench** | Software engineering benchmark for resolving real-world GitHub issues |
+| **hotpotqa** | Multi-hop question answering over Wikipedia |
+| **gsm8k** | Grade school math word problems with optional calculator tool |
+| **bfcl** | Berkeley Function Calling Leaderboard for evaluating tool-use capabilities |
+
+## Available Agents
+
+| Agent | Description |
+|-------|-------------|
+| **LiteLLM Tool Calling** | Generic tool-calling agent via LiteLLM |
+| **SmolAgents** | HuggingFace SmolAgents framework |
+| **OpenAI MCP** | OpenAI Responses API with MCP tools |
+| **Claude Code** | Anthropic Claude Code agent |
+| **Codex CLI** | OpenAI Codex CLI agent |
+| **Gemini CLI** | Google Gemini CLI agent |
+
+---
+
+## Dashboard
+
+
+
+```bash
+exgentic dashboard
+```
+
+---
+
+## Output Structure
+
+Each run creates its own directory under `outputs//`:
+
+```text
+outputs//
+├── results.json # Overall scores, costs, per-session statistics
+├── benchmark_results.json # Benchmark-specific aggregated results
+├── run/
+│ ├── config.json # Snapshot of benchmark and agent configuration
+│ ├── run.log # Main execution log
+│ └── warnings.log # Warnings during execution
+└── sessions//
+ ├── config.json # Session configuration
+ ├── results.json # Session results
+ ├── trajectory.jsonl # One JSON line per step (action + observation)
+ ├── agent/
+ │ └── agent.log # Agent execution log
+ └── benchmark/
+ ├── results.json # Benchmark-specific results
+ └── session.log # Benchmark session log
+```
+
+---
+
+## CLI Reference
+
+
+
+```bash
+# Discover
+exgentic list benchmarks
+exgentic list subsets --benchmark tau2
+exgentic list tasks --benchmark tau2 --subset retail --limit 5
+exgentic list agents
+exgentic install --benchmark tau2
+exgentic install --benchmark tau2 --docker
+exgentic install --benchmark tau2 --local
+exgentic uninstall --benchmark tau2
+
+# Run
+exgentic evaluate --benchmark tau2 --agent tool_calling --subset airline --num-tasks 10
+exgentic batch run --benchmark tau2 --agent tool_calling --subset airline --num-tasks 10
+
+# Inspect
+exgentic status --benchmark tau2 --agent tool_calling --subset airline --num-tasks 10
+exgentic preview --benchmark tau2 --agent tool_calling --subset airline --num-tasks 10
+exgentic results --benchmark tau2 --agent tool_calling --subset airline --num-tasks 10
+
+# Analyze
+exgentic compare --agents tool_calling openai --benchmark tau2
+
+# Explore
+exgentic dashboard
+```
+
+---
+
+## Advanced
+
+### Model Configuration
+
+```bash
+exgentic evaluate --benchmark tau2 --agent tool_calling --subset retail --num-tasks 2 \
+ --set agent.model.temperature=0.2
+```
+
+Supported fields: `temperature`, `top_p`, `max_tokens`, `reasoning_effort`, `num_retries`, `retry_after`, `retry_strategy`
+
+### Run Limits
+
+```bash
+exgentic evaluate --benchmark tau2 --agent tool_calling --subset retail --num-tasks 2 \
+ --max-steps 100 --max-actions 100
+```
+
+Sessions stop at either limit and record `limit_reached` status. Default: 100 for both.
+
+### HuggingFace
+
+Use HuggingFace models or run evaluations on HuggingFace Jobs. See [docs/huggingface.md](./docs/huggingface.md).
+
+---
+
+## How It Works
+
+To learn more about Exgentic's architecture and design, see our [arXiv paper](https://arxiv.org/abs/2602.22953).
+
+## Development
+
+For local development, editing, and contributing, see [DEVELOPMENT.md](./DEVELOPMENT.md).
+
+## Contributing
+
+We welcome issues and pull requests! See [CONTRIBUTING.md](./CONTRIBUTING.md) for guidelines.
+
+## Citing Exgentic
+
+```bibtex
+@misc{bandel2026generalagentevaluation,
+ title={General Agent Evaluation},
+ author={Elron Bandel and Asaf Yehudai and Lilach Eden and Yehoshua Sagron and Yotam Perlitz and Elad Venezian and Natalia Razinkov and Natan Ergas and Shlomit Shachor Ifergan and Segev Shlomov and Michal Jacovi and Leshem Choshen and Liat Ein-Dor and Yoav Katz and Michal Shmueli-Scheuer},
+ year={2026},
+ url={https://arxiv.org/abs/2602.22953},
+}
+```
+
+## License
+
+Apache License 2.0 — see [LICENSE](LICENSE).
+
+## Support
+
+For questions about this bundled copy,
+[open an issue](https://github.com/microsoft/Sico/issues) in Sico. For the
+original Exgentic project, use the
+[upstream issue tracker](https://github.com/Exgentic/exgentic/issues).
diff --git a/labs/AgentStream/exgentic/docs/README.md b/labs/AgentStream/exgentic/docs/README.md
new file mode 100644
index 00000000..c917d4f9
--- /dev/null
+++ b/labs/AgentStream/exgentic/docs/README.md
@@ -0,0 +1,38 @@
+# Exgentic Documentation
+
+Welcome to the Exgentic docs. Use the table below to find what you need.
+
+---
+
+## Using Exgentic
+
+| Document | Description |
+|----------|-------------|
+| [CLI Reference](./cli-reference.md) | Every command, flag, and environment variable |
+| [Python API](./python-api.md) | `evaluate()`, `execute()`, `aggregate()`, `status()`, `list_*()`, and all other library functions |
+| [Custom Models](./custom-models.md) | Use any LLM provider (OpenAI, Anthropic, Azure, Bedrock, Ollama, and more) via LiteLLM |
+| [Batch Runs](./batch.md) | Run parameter sweeps, manage large evaluations, export to CSV, publish to HuggingFace |
+| [Runners](./runners.md) | `direct`, `venv`, `docker` — isolation levels, configuration, Docker-in-Docker |
+| [Output Format](./output-format.md) | Schema for `results.json`, `trajectory.jsonl`, session results, and cost reports |
+| [Observers](./observers.md) | Hook into the evaluation lifecycle for custom logging, monitoring, and early stopping |
+
+## Extending Exgentic
+
+| Document | Description |
+|----------|-------------|
+| [Adding Agents](./adding-agents.md) | Write a new agent adapter — design principles, required methods, file layout, and validation checklist |
+| [Adding Benchmarks](./adding-benchmarks.md) | Write a new benchmark adapter — design principles, contract rules, and validation checklist |
+| [Replay Testing](./replay-testing.md) | Test benchmark and agent adapters end-to-end without API calls, using recorded sessions |
+
+## Observability
+
+| Document | Description |
+|----------|-------------|
+| [Quick Start](./observability/quickstart.md) | Set up OpenTelemetry tracing with Jaeger in five minutes |
+| [Semantic Conventions](./observability/semantic-conventions.md) | Full reference of every span and attribute Exgentic emits |
+
+## Other resources
+
+- [README.md](../README.md) — project overview, quick start, CLI reference, and available benchmarks/agents
+- [DEVELOPMENT.md](../DEVELOPMENT.md) — local setup, running tests, and linting
+- [CONTRIBUTING.md](../CONTRIBUTING.md) — contribution workflow, legal requirements, and PR guidelines
diff --git a/labs/AgentStream/exgentic/docs/adding-agents.md b/labs/AgentStream/exgentic/docs/adding-agents.md
new file mode 100644
index 00000000..2f6b3f2f
--- /dev/null
+++ b/labs/AgentStream/exgentic/docs/adding-agents.md
@@ -0,0 +1,384 @@
+# Adding Agents
+
+This document defines the agent design principles for Exgentic.
+
+It is intentionally opinionated. An agent adapter should not just "work"; it should cleanly separate configuration from execution, isolate heavy dependencies, and adapt to any benchmark contract without requiring the benchmark to change.
+
+Use these existing adapters as reference points:
+- `src/exgentic/agents/litellm_tool_calling/litellm_tool_calling_agent.py` + `instance.py` (split pattern)
+- `src/exgentic/agents/cli/claude/agent.py` (light pattern, single file)
+
+**Related docs:**
+[docs/](./README.md) · [Adding Benchmarks](./adding-benchmarks.md) · [Custom Models](./custom-models.md) · [Runners](./runners.md) · [Replay Testing](./replay-testing.md) · [CONTRIBUTING.md](../CONTRIBUTING.md)
+
+## Core Principle
+
+The agent adapts to the benchmark contract, not the other way around.
+
+That means:
+- the benchmark decides the task, context, actions, step flow, and scoring
+- the agent receives these through `_get_instance_kwargs()` and must work within them
+- the agent should not require benchmark modifications to function
+- the agent should not impose protocol-specific assumptions on the benchmark
+
+The default goal should be the thinnest possible agent wrapper.
+
+That means:
+- translate the benchmark's actions into whatever protocol your agent uses (tool calls, code generation, CLI commands)
+- do not reshape the benchmark contract to match your model's preferred format
+- keep the configuration surface small and explicit
+
+## Architecture
+
+Exgentic agents are split into two classes with distinct roles:
+
+### Agent (config, host-side)
+
+`Agent` is a lightweight Pydantic model that holds configuration. It lives on the host and is never sent into an isolated runner. It has no heavy dependencies.
+
+Responsibilities:
+- declare `display_name` and `slug_name` as `ClassVar[str]`
+- hold user-facing configuration fields (model name, max steps, feature flags)
+- implement `_get_instance_class()` to resolve the execution class
+- implement `_get_instance_kwargs()` to translate config + benchmark contract into constructor arguments
+- optionally override `setup()` for non-pip setup (Docker builds, npm installs)
+- optionally override `model_name` / `get_models_names()` for dashboard metadata
+
+### AgentInstance (execution, venv-side)
+
+`AgentInstance` is the execution class. It runs inside the runner (venv, Docker, or local) and may import heavy third-party libraries.
+
+Responsibilities:
+- implement `react(observation) -> Action | None` as the core decision loop
+- implement `close()` for resource cleanup
+- optionally override `start()` for initialization that happens after construction
+- optionally override `get_cost()` to report monetary cost
+
+The agent instance receives a single `session_id` in its constructor, which scopes all logs and artifacts. Additional kwargs come from `_get_instance_kwargs()`.
+
+## Key Pattern: Lazy Import for Dependency Isolation
+
+The `_get_instance_class()` classmethod must use a lazy import so that heavy dependencies are only loaded inside the runner environment, not on the host.
+
+```python
+@classmethod
+def _get_instance_class(cls):
+ from .instance import MyAgentInstance
+
+ return MyAgentInstance
+```
+
+This is the same pattern that `Benchmark._get_session_class()` uses. It ensures the host process never imports libraries like `litellm`, `smolagents`, `openai`, or any other agent-specific SDK.
+
+## When to Split Files
+
+**Split into separate files** when your agent depends on heavy third-party libraries:
+
+```
+src/exgentic/agents/my_agent/
+ __init__.py
+ my_agent.py # Agent subclass (light, no heavy imports)
+ instance.py # AgentInstance subclass (imports litellm, openai, etc.)
+ requirements.txt # Agent-specific pip dependencies
+ setup.sh # Optional non-pip setup
+ utils.py # Optional helpers
+```
+
+Examples: `litellm_tool_calling`, `smolagents`, `openai`
+
+**Keep everything in one file** when dependencies are light or already available in the base environment:
+
+```
+src/exgentic/agents/my_agent/
+ __init__.py
+ agent.py # Both Agent and AgentInstance in one file
+```
+
+Example: `cli/claude` (the instance class is in the same file because it only depends on stdlib and core Exgentic types)
+
+The rule is simple: if importing the instance class would pull in packages that are not in the base `exgentic` install, split the files.
+
+## Required Methods
+
+### On the Agent class
+
+#### `_get_instance_class()` (classmethod, abstract)
+
+Returns the `AgentInstance` subclass. Must use a lazy import.
+
+```python
+@classmethod
+def _get_instance_class(cls):
+ from .instance import MyAgentInstance
+
+ return MyAgentInstance
+```
+
+#### `_get_instance_kwargs()` (abstract)
+
+Translates the agent's configuration into constructor kwargs for the instance class. Task, context, and actions are passed separately via `start()`, not through the constructor.
+
+```python
+def _get_instance_kwargs(self, session_id: str) -> dict[str, Any]:
+ return {
+ "session_id": session_id,
+ "model": self.model,
+ "max_steps": self.max_steps,
+ }
+```
+
+The returned dict is passed directly to the instance class constructor. Every key must match a constructor parameter.
+
+### On the AgentInstance class
+
+#### `react(observation) -> Action | None` (abstract)
+
+The core decision loop. Receives an `Observation` (or `None` on the first call) and returns an `Action` to take, or `None` to signal that the agent is done.
+
+```python
+def react(self, observation: Observation | None) -> Action | None:
+ # Process observation, decide next action
+ # Return None when the agent decides it is finished
+ ...
+```
+
+#### `close()` (abstract)
+
+Cleanup resources. Called when the session ends, whether or not the agent finished normally.
+
+```python
+def close(self) -> None:
+ # Release connections, flush logs, etc.
+ pass
+```
+
+#### `start(task, context, actions)` (optional override)
+
+Called after construction but before the first `react()`. Receives the benchmark's task string, context dict, and list of action types. The base implementation stores these as `self.task`, `self.context`, and `self.actions`. Override to perform initialization that depends on these values (e.g., seeding a conversation with the task prompt).
+
+#### `get_cost()` (optional)
+
+Returns a `CostReport` with estimated monetary cost. Default returns an empty report. Override to track API costs.
+
+```python
+def get_cost(self) -> CostReport:
+ return self._cost_data
+```
+
+## Registration
+
+Every agent must be registered in `src/exgentic/interfaces/registry.py` in the `AGENTS` dict.
+
+```python
+AGENTS: dict[str, RegistryEntry] = {
+ # ...existing entries...
+ "my_agent": RegistryEntry(
+ slug_name="my_agent",
+ display_name="My Agent",
+ module="exgentic.agents.my_agent.my_agent",
+ attr="MyAgent",
+ kind="agent",
+ ),
+}
+```
+
+Requirements:
+- `slug_name` must match the `slug_name` ClassVar on the Agent class exactly
+- `display_name` must match the `display_name` ClassVar on the Agent class exactly
+- `module` is the dotted Python module path to the file containing the Agent class
+- `attr` is the class name within that module
+- `kind` must be `"agent"`
+
+The registry validates these constraints at load time. Mismatches will raise at startup.
+
+## Setup
+
+### `requirements.txt`
+
+List agent-specific pip dependencies. The runner installs these automatically into the isolated environment.
+
+```
+litellm>=1.50.0
+```
+
+Place the file in the agent's package directory. The `RunnerMixin` auto-discovers it by walking up from the module file.
+
+### `setup.sh`
+
+Optional script for non-pip setup. Runs after dependencies are installed.
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+# Build Docker images, install npm packages, download models, etc.
+```
+
+Place it next to the agent module. The `RunnerMixin` auto-discovers it.
+
+Both files are automatically found by the framework through `RunnerMixin.requirements_txt` and `RunnerMixin.setup_script`. No manual wiring is needed.
+
+## Recommended File Structure
+
+### Split pattern (heavy deps)
+
+```
+src/exgentic/agents/my_agent/
+ __init__.py
+ my_agent.py # Agent subclass
+ instance.py # AgentInstance subclass
+ requirements.txt # e.g., litellm>=1.50.0
+ setup.sh # optional
+ utils.py # optional helpers
+```
+
+**my_agent.py** (host-side, no heavy imports):
+
+```python
+from __future__ import annotations
+
+from typing import Any, ClassVar
+
+from ...core.agent import Agent
+from ...core.types import ActionType, ModelSettings
+
+
+class MyAgent(Agent):
+ display_name: ClassVar[str] = "My Agent"
+ slug_name: ClassVar[str] = "my_agent"
+
+ model: str = "gpt-4o"
+ max_steps: int = 100
+
+ @classmethod
+ def _get_instance_class(cls):
+ from .instance import MyAgentInstance
+
+ return MyAgentInstance
+
+ @property
+ def model_name(self) -> str:
+ return self.model
+
+ def _get_instance_kwargs(self, session_id: str) -> dict[str, Any]:
+ return {
+ "session_id": session_id,
+ "model": self.model,
+ "max_steps": self.max_steps,
+ }
+```
+
+**instance.py** (runner-side, may import heavy libs):
+
+```python
+from __future__ import annotations
+
+from typing import Any, Optional
+
+import some_heavy_library # only loaded inside the runner
+
+from ...core.agent_instance import AgentInstance
+from ...core.types import Action, ActionType, Observation
+
+
+class MyAgentInstance(AgentInstance):
+ def __init__(
+ self,
+ session_id: str,
+ task: str,
+ context: dict[str, Any],
+ actions: list[ActionType],
+ model: str,
+ max_steps: int,
+ ):
+ super().__init__(session_id)
+ self.task = task
+ self.context = context
+ self.actions = actions
+ self.model = model
+ self.max_steps = max_steps
+ self._step_count = 0
+
+ def react(self, observation: Optional[Observation]) -> Optional[Action]:
+ self._step_count += 1
+ if self._step_count > self.max_steps:
+ return None
+ # Agent decision logic here
+ ...
+
+ def close(self) -> None:
+ pass
+```
+
+### Light pattern (no heavy deps)
+
+```
+src/exgentic/agents/my_agent/
+ __init__.py
+ agent.py # Both Agent and AgentInstance
+```
+
+Keep both classes in a single file when the instance has no third-party imports beyond what exgentic already provides.
+
+## Validation Checklist
+
+Before opening a PR for a new agent, validate all of the following.
+
+### Contract validation
+
+- Agent class declares `display_name` and `slug_name` as `ClassVar[str]`
+- `_get_instance_class()` uses a lazy import
+- `_get_instance_kwargs()` returns a dict whose keys match the instance constructor
+- The instance implements `react()` and `close()`
+- The instance calls `super().__init__(session_id)` in its constructor
+
+### Dependency isolation validation
+
+- The Agent file does not import heavy third-party libraries at module level
+- Heavy imports only appear inside `_get_instance_class()` or in the instance module
+- `requirements.txt` lists all agent-specific dependencies
+
+### Registry validation
+
+- `slug_name` in the registry entry matches the class `slug_name` exactly
+- `display_name` in the registry entry matches the class `display_name` exactly
+- `module` path resolves to the correct file
+- `attr` matches the Agent class name
+- `kind` is `"agent"`
+
+### Functional validation
+
+- Agent is discoverable through the registry (`load_agent("my_agent")` succeeds)
+- Agent works with at least one benchmark end to end
+- `react()` correctly returns `None` when the agent decides it is done
+- `close()` does not raise
+- `get_cost()` returns a valid `CostReport`
+
+### Quality validation
+
+- `py_compile` passes for all changed Python files
+- `pre-commit` passes for changed files
+- `git diff --check` passes
+
+## Practical Rule of Thumb
+
+When in doubt, ask:
+
+1. Does the agent adapt to the benchmark, or does it require the benchmark to change?
+2. Are heavy dependencies isolated behind a lazy import?
+3. Is the Agent file importable without installing agent-specific packages?
+4. Does `_get_instance_kwargs()` faithfully pass the benchmark contract through?
+5. Will this agent work with benchmarks that have very different action spaces?
+6. Is the configuration surface minimal and explicit?
+
+If the answer to any of those is no, the adapter is probably too coupled or too leaky.
+
+---
+
+## See also
+
+- [Adding Benchmarks](./adding-benchmarks.md) — the other side of the contract
+- [Custom Models](./custom-models.md) — configuring LLM providers and sampling parameters for the `tool_calling` agent
+- [Runners](./runners.md) — how setup.sh and requirements.txt are discovered and executed
+- [Replay Testing](./replay-testing.md) — write end-to-end tests for your agent without API calls
+- [CONTRIBUTING.md](../CONTRIBUTING.md) — PR workflow and legal requirements
+- [docs/](./README.md) — documentation index
diff --git a/labs/AgentStream/exgentic/docs/adding-benchmarks.md b/labs/AgentStream/exgentic/docs/adding-benchmarks.md
new file mode 100644
index 00000000..4fd56287
--- /dev/null
+++ b/labs/AgentStream/exgentic/docs/adding-benchmarks.md
@@ -0,0 +1,364 @@
+# Adding Benchmarks
+
+This document defines the benchmark design principles for Exgentic.
+
+It is intentionally opinionated. A benchmark adapter should not just "work"; it should preserve the benchmark's meaning while still fitting Exgentic's agent abstraction cleanly.
+
+Use these existing adapters as reference points:
+- `src/exgentic/benchmarks/tau2/tau2_benchmark.py`
+- `src/exgentic/benchmarks/bfcl/bfcl_benchmark.py`
+
+**Related docs:**
+[docs/](./README.md) · [Adding Agents](./adding-agents.md) · [Runners](./runners.md) · [Replay Testing](./replay-testing.md) · [Output Format](./output-format.md) · [CONTRIBUTING.md](../CONTRIBUTING.md)
+
+## Core Principle
+
+The benchmark owns the contract.
+
+That means the benchmark decides:
+- what the task is
+- what context the agent receives
+- what actions exist
+- how steps progress
+- when a session is finished
+- how scoring works
+
+The agent should adapt to the benchmark contract through Exgentic's normal interfaces. The benchmark should not be shaped around one specific model protocol.
+
+The default goal should be the thinnest possible benchmark wrapper.
+
+That means:
+- reuse the source benchmark wherever possible
+- add only the translation layers that are actually necessary
+- avoid reimplementing benchmark logic unless there is a clear reason
+- avoid introducing runtime behavior that exists only to satisfy one agent or one model protocol
+
+The target is simple:
+- make the benchmark accessible to any Exgentic agent
+- while adding the minimum adapter surface necessary
+- and without clashing with agent-specific assumptions
+
+## Principles
+
+### 1. Keep the agent-facing contract protocol-agnostic
+
+Do not define a benchmark in terms of OpenAI tool calls, raw assistant messages, or any other provider-specific response format.
+
+Define it in terms of:
+- task semantics
+- available actions
+- observations
+- finish conditions
+- score
+
+Protocol-specific translation belongs in adapters, not in the benchmark contract.
+
+Bad:
+- "The model must return all tool calls in one assistant message."
+
+Good:
+- "The task is complete when the required actions have been taken and the benchmark-specific finish condition is met."
+
+### 2. The task should be the real task
+
+`task` should contain the actual task the agent is meant to solve.
+
+Do not wrap the task in fake chat scaffolding unless that scaffolding is genuinely part of the benchmark.
+
+If the benchmark is not about user interaction, do not invent a chat conversation just to make it look conversational.
+
+Bad:
+- synthetic "user" messages when the benchmark is not actually testing user interaction
+- generic wrapper prompts replacing the real benchmark task
+
+Good:
+- the benchmark prompt itself is the `task`
+
+### 3. Context should contain only what the agent should know
+
+`context` is not a metadata dump.
+
+It should contain only information that is necessary for the agent to behave correctly on the task.
+
+Keep internal benchmark metadata out of `context`, including:
+- subset names
+- source dataset ids
+- registry information
+- adapter implementation details
+
+Good context:
+- policy text
+- execution constraints
+- information the agent genuinely needs to act correctly
+
+Bad context:
+- `"subset": "live_parallel_multiple"`
+- `"benchmark": "bfcl"`
+
+### 4. Actions should represent semantic operations
+
+Actions are the benchmark's action space.
+
+Name and describe them in terms of what they do, not in terms of a transport protocol.
+
+Prefer "actions" over protocol-specific terms like "tool calls" in benchmark-facing language, because not all agents consume or produce actions through the same protocol.
+
+If the source benchmark exposes functions, commands, or tools, translate those into Exgentic actions at the boundary.
+
+### 5. Use `finish` only as part of the benchmark contract
+
+`finish` is valid when the benchmark needs an explicit end-of-step or end-of-task signal.
+
+It should exist because the benchmark contract needs it, not because a specific model API needs it.
+
+Use it when:
+- the benchmark has multiple steps or turns and needs an explicit transition point
+- the benchmark needs a clear "done with this step" signal
+- the benchmark should allow completion without another normal action
+
+Do not force `finish` into a benchmark if the source benchmark's semantics are cleaner without it.
+
+### 6. Distinguish execution modes by contract, not by protocol
+
+If a benchmark has single-turn, live, or multi-turn variants, define those as execution contracts.
+
+The important differences are things like:
+- whether more steps may follow
+- whether the task ends after the current finish
+- whether action outputs affect later state
+- whether the benchmark continues after a step completes
+
+Do not define the mode in terms of how many assistant messages or tool-call payloads a model is allowed to emit.
+
+Important:
+- single-turn does not necessarily mean a single action
+- multi-action single-turn tasks are valid
+- the distinction is about step structure, not about one specific model protocol
+
+### 7. Action outputs must be honest
+
+If the benchmark can produce real execution outputs, use them.
+
+If it cannot, do not fabricate realistic outputs that imply more runtime semantics than actually exist.
+
+Be explicit in the contract when actions are only being recorded rather than executed.
+
+Good:
+- real execution results when the source benchmark exposes an official executor
+- `Action recorded.` when there is no real runtime execution for that task family
+
+Bad:
+- made-up outputs that look like real environment state changes when none were actually computed
+
+### 8. Reuse external harnesses as the source of truth where possible
+
+When adapting an external benchmark, prefer to reuse:
+- dataset loading
+- official assets
+- ground-truth files
+- official checkers or scorers
+- official execution helpers
+
+Avoid copying large chunks of benchmark logic into Exgentic if the source repository already provides them.
+
+But there is an important boundary:
+- external harnesses should be the source of truth for benchmark assets and scoring
+- they should not automatically own the Exgentic runtime contract
+
+If the external harness assumes a model-specific interaction pattern, Exgentic should usually keep its own runtime and bridge to the harness at load/score time instead.
+
+When choosing between two valid integrations, prefer the thinner one.
+
+Use the more complex approach only when the thinner one would:
+- distort benchmark meaning
+- hard-code one agent's assumptions
+- or force Exgentic to own logic that should stay with the source benchmark
+
+### 9. Be explicit about what is official and what is adapted
+
+If the adapter preserves official scoring but changes runtime behavior, document that clearly.
+
+If some subsets use official execution while others only use official scoring, document that too.
+
+Do not imply full equivalence when the integration is intentionally more abstract than the source benchmark.
+
+For each benchmark adapter, it should be easy to answer:
+- What comes directly from the source benchmark?
+- What is adapted by Exgentic?
+- What is exact?
+- What is approximate?
+
+### 10. Success, failure, and error must stay distinct
+
+Finished benchmark failures are not the same as runtime errors.
+
+The adapter should keep these states separate:
+- success: benchmark completed and passed
+- unsuccessful: benchmark completed and failed
+- unfinished: benchmark did not complete
+- error: adapter or runtime failure prevented a proper benchmark result
+
+Do not swallow real errors and report them as ordinary failures.
+
+If an exception happens, record it explicitly in session metadata.
+
+### 11. The benchmark should work for many agents, not just one
+
+A benchmark adapter should not depend on modifying one particular agent implementation.
+
+Prefer to build benchmark logic around Exgentic's shared abstractions:
+- `task`
+- `context`
+- `actions`
+- observations
+- `Session.start()`
+- `Session.step()`
+- `Session.done()`
+- `Session.score()`
+
+If the adapter only works because one agent has special behavior, the adapter is too coupled.
+
+### 12. Keep setup, runtime, and registration separate
+
+A well-structured benchmark adapter usually has three separate concerns:
+
+1. Setup
+- external checkout or installation
+- pinned dependencies
+- benchmark-specific environment preparation
+
+2. Runtime
+- session logic
+- task loading
+- action translation
+- scoring
+
+3. Registration
+- registry entry
+- subset listing
+- CLI discoverability
+
+Do not mix setup logic directly into the runtime path when it can be handled once in `setup.sh`.
+
+### 13. The main benchmark file must not import external dependencies
+
+The main benchmark file (`_benchmark.py`) defines the `Benchmark` subclass that Exgentic loads in the host process. This file **must be importable without any benchmark-specific dependencies installed**.
+
+External dependencies (benchmark harnesses, datasets, ML libraries, etc.) belong in **separate files** that are only loaded inside the runner subprocess through `_get_evaluator_class()` and `_get_session_class()`.
+
+**Rule:** The benchmark class file may only import from:
+- Python standard library
+- `pydantic`
+- `exgentic` core modules
+
+All other imports must live in evaluator/session files that are accessed through the class getters.
+
+**Why:** Exgentic loads the benchmark class in the host process to read configuration (runner type, evaluator/session class names, kwargs). The actual benchmark execution happens inside an isolated runner (venv or Docker). If the main file imports heavy dependencies, the host process fails when those deps are only installed inside the runner environment.
+
+Bad:
+```python
+# _benchmark.py
+from some_harness import HarnessRunner # ← breaks host import
+
+class MyBenchmark(Benchmark):
+ ...
+```
+
+Good:
+```python
+# _benchmark.py — no external deps
+class MyBenchmark(Benchmark):
+ def _get_evaluator_class(self):
+ from ._eval import MyEvaluator # loaded inside runner
+ return MyEvaluator
+
+# _eval.py — external deps are fine here
+from some_harness import HarnessRunner # ← only loaded in runner subprocess
+```
+
+## Required Structure
+
+For a benchmark package under `src/exgentic/benchmarks//`:
+
+- `_benchmark.py` **(required)**
+ - `Benchmark` subclass only
+ - no external dependency imports
+ - `_get_evaluator_class()` and `_get_session_class()` return classes from other files
+- `_eval.py` or `_session.py` **(required if benchmark has external deps)**
+ - evaluator, session, and runtime logic
+ - may import external dependencies at module level
+ - only loaded inside the runner subprocess
+- `setup.sh`
+ - benchmark installation/bootstrap
+- optional shim module
+ - thin import boundary around an external harness
+- optional helper modules
+ - action translation, scoring helpers, data parsing
+
+Then register it in:
+- `src/exgentic/interfaces/registry.py`
+
+## Validation Checklist
+
+Before opening a PR for a new benchmark, validate all of the following.
+
+### Contract validation
+
+- `task` is the actual task, not fake wrapper chat
+- `context` contains only agent-relevant information
+- actions are semantically named
+- `finish` exists only if the benchmark contract needs it
+- success/failure/error semantics are distinct
+
+### Import validation
+
+- the main benchmark file (`_benchmark.py`) imports **no external dependencies**
+- `_get_evaluator_class()` and `_get_session_class()` load from separate files
+- `python -c "from exgentic.benchmarks.._benchmark import "` works without deps installed
+
+### Functional validation
+
+- benchmark is discoverable through the registry
+- subsets list correctly
+- tasks list correctly
+- setup script works from a clean environment
+- at least one happy-path task works end to end
+- benchmark works with the default venv runner (not just direct)
+- at least one failure-path task is represented correctly
+- adapter errors surface as errors, not silent failures
+
+### Source-of-truth validation
+
+- official assets are reused where possible
+- official scoring is reused where possible
+- any remaining deviations from the source benchmark are documented explicitly
+
+### Quality validation
+
+- `py_compile` passes for changed Python files
+- `pre-commit` passes for changed files
+- `git diff --check` passes
+
+## Practical Rule Of Thumb
+
+When in doubt, ask:
+
+1. Is this benchmark contract describing the task, or just mirroring one model API?
+2. Is this information something the agent should truly know?
+3. Is this the thinnest adapter that still preserves the benchmark's meaning?
+4. Am I reusing the source benchmark where it helps, without letting it dictate the wrong runtime shape?
+5. Are the benchmark outputs honest about what was actually executed?
+6. Will this adapter still make sense for a very different kind of Exgentic agent?
+
+If the answer to any of those is no, the adapter is probably too coupled or too misleading.
+
+---
+
+## See also
+
+- [Adding Agents](./adding-agents.md) — the other side of the contract
+- [Runners](./runners.md) — how setup.sh and requirements.txt are discovered and executed
+- [Replay Testing](./replay-testing.md) — write end-to-end tests for your benchmark without API calls
+- [Output Format](./output-format.md) — trajectory.jsonl and results.json schemas
+- [CONTRIBUTING.md](../CONTRIBUTING.md) — PR workflow and legal requirements
+- [docs/](./README.md) — documentation index
diff --git a/labs/AgentStream/exgentic/docs/batch.md b/labs/AgentStream/exgentic/docs/batch.md
new file mode 100644
index 00000000..86e19700
--- /dev/null
+++ b/labs/AgentStream/exgentic/docs/batch.md
@@ -0,0 +1,269 @@
+# Batch Runs
+
+The `batch` commands let you manage large evaluations across multiple configurations — parameter sweeps, multi-benchmark comparisons, re-runs of failed sessions — without writing orchestration scripts.
+
+**Related docs:**
+[docs/](./README.md) · [CLI Reference](./cli-reference.md) · [Python API](./python-api.md) · [Output Format](./output-format.md)
+
+---
+
+## When to use batch vs evaluate
+
+| Scenario | Command |
+|----------|---------|
+| Single benchmark run | `exgentic evaluate` |
+| Multiple models on one benchmark | `batch evaluate` with config files |
+| Re-run only failed sessions | `batch evaluate` (skips completed by default) |
+| Sweep temperature/model grids | `batch evaluate` + `batch patch` |
+| Publish results to HuggingFace | `batch publish` |
+| Export results to CSV | `batch extract` |
+
+---
+
+## Config files
+
+Every batch command operates on **config files** — JSON files that describe a run. There are two kinds.
+
+### RunConfig
+
+Describes a full multi-task run.
+
+```json
+{
+ "benchmark": "tau2",
+ "agent": "tool_calling",
+ "subset": "retail",
+ "num_tasks": 10,
+ "model": "gpt-4o",
+ "benchmark_kwargs": {
+ "user_simulator_model": "gpt-4o"
+ },
+ "agent_kwargs": {
+ "model_settings": {
+ "temperature": 0.2
+ }
+ },
+ "output_dir": "./outputs",
+ "max_steps": 100,
+ "max_actions": 100
+}
+```
+
+### SessionConfig
+
+Describes a single task. Used when you need per-task control or when replaying individual sessions.
+
+```json
+{
+ "benchmark": "tau2",
+ "agent": "tool_calling",
+ "task_id": "retail_1",
+ "subset": "retail",
+ "model": "gpt-4o",
+ "output_dir": "./outputs"
+}
+```
+
+### Full field reference
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `benchmark` | string | required | Benchmark slug (e.g. `tau2`) |
+| `agent` | string | required | Agent slug (e.g. `tool_calling`) |
+| `subset` | string | null | Benchmark subset |
+| `task_ids` | list[string] | null | Explicit task IDs to run |
+| `num_tasks` | int | null | Number of tasks (randomly sampled if task_ids not set) |
+| `task_id` | string | required (SessionConfig) | Single task ID |
+| `model` | string | null | Model override |
+| `output_dir` | string | `./outputs` | Where to write results |
+| `cache_dir` | string | null | Cache directory |
+| `run_id` | string | auto | Deterministic ID derived from config |
+| `max_steps` | int | 100 | Steps per session |
+| `max_actions` | int | 100 | Actions per session |
+| `max_workers` | int | null | Parallel session workers |
+| `overwrite_sessions` | bool | false | Re-run already-completed sessions |
+| `benchmark_kwargs` | object | null | Extra kwargs passed to the benchmark |
+| `agent_kwargs` | object | null | Extra kwargs passed to the agent |
+
+---
+
+## Commands
+
+All batch commands accept one or more `--config` flags, each taking a file path or a glob pattern.
+
+```bash
+exgentic batch --config path/to/config.json
+exgentic batch --config "configs/*.json"
+exgentic batch --config configs/run1.json --config configs/run2.json
+```
+
+---
+
+### batch evaluate
+
+Run all configs sequentially, executing sessions and aggregating results.
+
+```bash
+exgentic batch evaluate --config "configs/*.json"
+```
+
+Already-completed sessions are skipped unless `overwrite_sessions` is true in the config. This makes it safe to re-run after partial failures — only missing or failed sessions are executed.
+
+---
+
+### batch execute
+
+Same as `batch evaluate` but skips the aggregation step. Use this when you want to run sessions and aggregate later.
+
+```bash
+exgentic batch execute --config "configs/*.json"
+exgentic batch aggregate --config "configs/*.json" # aggregate afterwards
+```
+
+---
+
+### batch aggregate
+
+Aggregate results from already-completed sessions without running anything.
+
+```bash
+exgentic batch aggregate --config "configs/*.json"
+```
+
+Useful when you have sessions from a previous run and want to recompute scores.
+
+---
+
+### batch status
+
+Print a status table showing completion state for each config.
+
+```bash
+exgentic batch status --config "configs/*.json"
+```
+
+---
+
+### batch prepare
+
+Write session config files to disk without executing. Creates the session directory structure so you can inspect or modify configs before running.
+
+```bash
+exgentic batch prepare --config run.json
+exgentic batch prepare --config run.json --overwrite # overwrite existing session configs
+```
+
+---
+
+### batch patch
+
+Modify existing run or session config files in bulk using dotted-key notation.
+
+```bash
+# Preview what would change
+exgentic batch patch --config "configs/*.json" \
+ --set model=gpt-4o \
+ --dry-run
+
+# Apply changes
+exgentic batch patch --config "configs/*.json" \
+ --set model=gpt-4o \
+ --set agent_kwargs.model_settings.temperature=0.2 \
+ --apply
+```
+
+Dotted paths are resolved into nested dicts. Values are parsed as JSON first; if that fails, treated as strings. This lets you do sweeps:
+
+```bash
+# Change model across a whole grid of configs
+exgentic batch patch --config "sweep_*.json" --set model=claude-3-5-sonnet-20241022 --apply
+```
+
+---
+
+### batch extract
+
+Export results from multiple runs into a single CSV file.
+
+```bash
+exgentic batch extract --config "configs/*.json" --output results.csv
+exgentic batch extract --config "configs/*.json" --output - # print to stdout
+```
+
+Each row is one run. Columns include all `RunResults` fields (see [Output Format](./output-format.md)).
+
+---
+
+### batch publish
+
+Push results to a [HuggingFace dataset](https://huggingface.co/docs/datasets/).
+
+```bash
+exgentic batch publish \
+ --config "configs/*.json" \
+ --repo Exgentic/open-agent-leaderboard-results \
+ --append
+```
+
+Flags:
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `--repo` | required | HuggingFace dataset repo ID |
+| `--append` / `--overwrite` | `--append` | Append to or replace the existing dataset |
+| `--private` / `--public` | `--private` | Dataset visibility |
+
+Deduplication: when appending, existing rows with the same `(benchmark, agent, model)` triple are replaced. New combinations are appended.
+
+Requires the `datasets` package (`pip install datasets`) and a HuggingFace token with write access:
+
+```bash
+huggingface-cli login
+# or
+export HF_TOKEN=hf_...
+```
+
+---
+
+## Typical workflows
+
+### Parameter sweep
+
+```bash
+# Create one config per model
+for model in gpt-4o claude-3-5-sonnet-20241022 gemini-2.0-flash; do
+ cp base_config.json "configs/${model}.json"
+ exgentic batch patch --config "configs/${model}.json" --set model=${model} --apply
+done
+
+# Run all
+exgentic batch evaluate --config "configs/*.json"
+
+# Export to CSV
+exgentic batch extract --config "configs/*.json" --output sweep_results.csv
+```
+
+### Resume after partial failure
+
+```bash
+# Just re-run — completed sessions are skipped automatically
+exgentic batch evaluate --config "configs/*.json"
+```
+
+### Separate execute from aggregate
+
+```bash
+# Run sessions in parallel across machines, then aggregate centrally
+exgentic batch execute --config "configs/*.json"
+# ... copy outputs to aggregation machine ...
+exgentic batch aggregate --config "configs/*.json"
+```
+
+---
+
+## See also
+
+- [CLI Reference](./cli-reference.md) — full flag reference for all commands
+- [Output Format](./output-format.md) — RunResults schema, what batch extract produces
+- [Python API](./python-api.md) — programmatic equivalents: `evaluate()`, `execute()`, `aggregate()`
+- [docs/](./README.md) — documentation index
diff --git a/labs/AgentStream/exgentic/docs/cli-reference.md b/labs/AgentStream/exgentic/docs/cli-reference.md
new file mode 100644
index 00000000..2c9235ba
--- /dev/null
+++ b/labs/AgentStream/exgentic/docs/cli-reference.md
@@ -0,0 +1,323 @@
+# CLI Reference
+
+Complete reference for all `exgentic` CLI commands.
+
+**Related docs:**
+[docs/](./README.md) · [Python API](./python-api.md) · [Batch Runs](./batch.md) · [Custom Models](./custom-models.md) · [Output Format](./output-format.md)
+
+---
+
+## Global flags
+
+| Flag | Description |
+|------|-------------|
+| `--debug` | Enable debug logging |
+| `--help` | Show help for any command |
+
+---
+
+## Discovery
+
+### list benchmarks
+
+List all available benchmarks.
+
+```bash
+exgentic list benchmarks
+```
+
+### list agents
+
+List all available agents.
+
+```bash
+exgentic list agents
+```
+
+### list subsets
+
+List subsets for a benchmark.
+
+```bash
+exgentic list subsets --benchmark tau2
+```
+
+### list tasks
+
+List task IDs for a benchmark (or subset).
+
+```bash
+exgentic list tasks --benchmark tau2 --subset retail
+exgentic list tasks --benchmark tau2 --subset retail --limit 20
+```
+
+| Flag | Description |
+|------|-------------|
+| `--benchmark` | Benchmark slug (required) |
+| `--subset` | Subset name |
+| `--limit` | Maximum tasks to show |
+
+---
+
+## install
+
+Install a benchmark's or agent's dependencies (default: isolated venv).
+
+```bash
+exgentic install --benchmark tau2 # install deps + data (default: venv)
+exgentic install --agent tool_calling
+exgentic install --benchmark tau2 --force # reinstall even if already set up
+exgentic install --benchmark tau2 --docker # build Docker image
+exgentic install --benchmark tau2 --local # install into local environment
+```
+
+| Flag | Description |
+|------|-------------|
+| `--benchmark` | Benchmark slug |
+| `--agent` | Agent slug |
+| `--force` | Force reinstall |
+| `--docker` | Build a Docker image |
+| `--local` | Install into the local environment instead of an isolated venv |
+
+See [Runners](./runners.md) for details on runner types.
+
+---
+
+## uninstall
+
+Remove an installed benchmark's or agent's environment.
+
+```bash
+exgentic uninstall --benchmark tau2
+exgentic uninstall --agent tool_calling
+```
+
+| Flag | Description |
+|------|-------------|
+| `--benchmark` | Benchmark slug |
+| `--agent` | Agent slug |
+
+---
+
+## setup (deprecated)
+
+> **Deprecated:** `exgentic setup` is an alias for `exgentic install` and will be removed in a future release. Use `install`/`uninstall` instead.
+
+---
+
+## evaluate
+
+Run an evaluation end-to-end: execute sessions and aggregate results.
+
+```bash
+exgentic evaluate \
+ --benchmark tau2 \
+ --agent tool_calling \
+ --subset retail \
+ --num-tasks 10 \
+ --model gpt-4o \
+ --set benchmark.user_simulator_model="gpt-4o"
+```
+
+| Flag | Description |
+|------|-------------|
+| `--benchmark` | Benchmark slug (required) |
+| `--agent` | Agent slug (required) |
+| `--subset` | Benchmark subset |
+| `--task` | One or more specific task IDs (repeatable) |
+| `--num-tasks` | Number of tasks to run |
+| `--model` | Model override |
+| `--max-steps` | Steps per session (default: 100) |
+| `--max-actions` | Actions per session (default: 100) |
+| `--max-workers` | Parallel session workers |
+| `--overwrite` | Re-run already-completed sessions |
+| `--output-dir` | Results output directory (default: `./outputs`) |
+| `--run-id` | Override the auto-generated run ID |
+| `--set KEY=VALUE` | Override any config field (repeatable) |
+| `--debug` | Enable debug logging |
+
+### --set syntax
+
+`--set` accepts dotted key paths and JSON-compatible values:
+
+```bash
+# Benchmark kwargs
+--set benchmark.user_simulator_model="gpt-4o"
+--set benchmark.runner=venv
+
+# Agent kwargs
+--set agent.max_steps=200
+
+# Model settings
+--set agent.model.temperature=0.2
+--set agent.model.max_tokens=4096
+--set agent.model.top_p=0.9
+--set agent.model.reasoning_effort=high
+--set agent.model.num_retries=3
+--set agent.model.retry_after=1.0
+--set agent.model.retry_strategy=constant
+```
+
+---
+
+## status
+
+Show the execution status of a run (how many sessions are done, running, missing).
+
+```bash
+exgentic status --benchmark tau2 --agent tool_calling --subset retail --num-tasks 10
+```
+
+Accepts the same flags as `evaluate`.
+
+---
+
+## preview
+
+Show which tasks would run without executing anything.
+
+```bash
+exgentic preview --benchmark tau2 --agent tool_calling --subset retail --num-tasks 10
+```
+
+Prints a plan showing which sessions would be new, which already exist, and which are currently running.
+
+---
+
+## results
+
+Load and display results from a completed run.
+
+```bash
+exgentic results --benchmark tau2 --agent tool_calling --subset retail --num-tasks 10
+```
+
+Reads `results.json` from the run directory. Accepts the same config flags as `evaluate`.
+
+See [Output Format](./output-format.md) for the full results schema.
+
+---
+
+## compare
+
+Statistical comparison between two run configurations.
+
+```bash
+exgentic compare \
+ --agents tool_calling openai_solo \
+ --benchmark tau2 \
+ --subset retail \
+ --num-tasks 50
+```
+
+Runs a Breslow-Day homogeneity test across subsets and reports whether the difference between agents is statistically significant.
+
+Requires the `analysis` extra:
+
+```bash
+pip install "exgentic[analysis]"
+```
+
+---
+
+## analyze
+
+Generate comparison plots for multiple benchmarks or agents.
+
+```bash
+exgentic analyze \
+ --agents tool_calling openai_solo \
+ --benchmarks tau2 gsm8k \
+ --output report.png
+```
+
+Requires the `analysis` extra:
+
+```bash
+pip install "exgentic[analysis]"
+```
+
+---
+
+## dashboard
+
+Launch the interactive web dashboard.
+
+```bash
+exgentic dashboard
+```
+
+Opens a NiceGUI interface for exploring runs, browsing session trajectories, and monitoring live evaluations.
+
+---
+
+## batch
+
+All batch subcommands. See [Batch Runs](./batch.md) for full documentation.
+
+```bash
+exgentic batch evaluate --config "configs/*.json"
+exgentic batch execute --config "configs/*.json"
+exgentic batch aggregate --config "configs/*.json"
+exgentic batch status --config "configs/*.json"
+exgentic batch prepare --config run.json [--overwrite]
+exgentic batch patch --config "configs/*.json" --set key=value [--apply | --dry-run]
+exgentic batch extract --config "configs/*.json" --output results.csv
+exgentic batch publish --config "configs/*.json" --repo org/dataset [--append | --overwrite] [--private | --public]
+```
+
+---
+
+## Environment variables
+
+Exgentic reads the following environment variables.
+
+### Exgentic settings
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `EXGENTIC_LOG_LEVEL` | `INFO` | Log level: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL` |
+| `EXGENTIC_CACHE_DIR` | `.exgentic` | Cache directory for venvs and setup state |
+| `EXGENTIC_DOTENV_PATH` | `.env` | Path to `.env` file loaded automatically |
+| `EXGENTIC_OTEL_ENABLED` | `false` | Enable OpenTelemetry tracing |
+| `EXGENTIC_OTEL_RECORD_CONTENT` | `false` | Include prompts/responses in traces (opt-in) |
+| `EXGENTIC_LITELLM_CACHING` | `true` | Enable LiteLLM response caching |
+| `EXGENTIC_LITELLM_CACHE_DIR` | `~/.cache/exgentic/litellm` | LiteLLM cache directory |
+| `EXGENTIC_LITELLM_LOG_LEVEL` | `WARNING` | LiteLLM internal log level |
+
+### LLM provider credentials
+
+| Variable | Provider |
+|----------|----------|
+| `OPENAI_API_KEY` | OpenAI |
+| `ANTHROPIC_API_KEY` | Anthropic |
+| `AZURE_API_KEY` | Azure OpenAI |
+| `AZURE_API_BASE` | Azure OpenAI endpoint |
+| `AZURE_API_VERSION` | Azure OpenAI API version |
+| `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` | AWS Bedrock |
+| `AWS_REGION_NAME` | AWS Bedrock region |
+| `VERTEXAI_PROJECT` / `VERTEXAI_LOCATION` | Google Vertex AI |
+| `OPENAI_API_BASE` | Custom OpenAI-compatible endpoint |
+
+See [Custom Models](./custom-models.md) for full provider setup instructions.
+
+### OpenTelemetry
+
+| Variable | Description |
+|----------|-------------|
+| `OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP collector endpoint |
+| `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf` or `grpc` |
+
+See [Observability Quick Start](./observability/quickstart.md) for tracing setup.
+
+---
+
+## See also
+
+- [Python API](./python-api.md) — programmatic equivalents of all CLI commands
+- [Batch Runs](./batch.md) — detailed guide for batch commands
+- [Custom Models](./custom-models.md) — LLM provider and `--set agent.model.*` reference
+- [Runners](./runners.md) — `--set benchmark.runner=*` options
+- [Output Format](./output-format.md) — what `results` and `extract` produce
+- [Observability Quick Start](./observability/quickstart.md) — tracing setup
+- [docs/](./README.md) — documentation index
diff --git a/labs/AgentStream/exgentic/docs/custom-models.md b/labs/AgentStream/exgentic/docs/custom-models.md
new file mode 100644
index 00000000..f81be751
--- /dev/null
+++ b/labs/AgentStream/exgentic/docs/custom-models.md
@@ -0,0 +1,250 @@
+# Custom Models
+
+Exgentic routes all LLM calls through [LiteLLM](https://docs.litellm.ai/), which means any provider or deployment LiteLLM supports works out of the box — no code changes required. You pick the model, supply credentials, and optionally tune sampling parameters.
+
+**Related docs:**
+[docs/](./README.md) · [CLI Reference](./cli-reference.md) · [Python API](./python-api.md) · [Adding Agents](./adding-agents.md) · [Observability Quick Start](./observability/quickstart.md)
+
+---
+
+## Model string format
+
+The `--model` flag (and the `model` parameter in the Python API) accepts any model string that LiteLLM recognises. The general pattern is:
+
+```
+/
+```
+
+For OpenAI-native models the provider prefix is optional:
+
+```bash
+# These are equivalent
+--model gpt-4o
+--model openai/gpt-4o
+```
+
+For every other provider the prefix is required. See the provider examples below.
+
+---
+
+## Supported providers
+
+### OpenAI
+
+```bash
+exgentic evaluate --benchmark gsm8k --agent tool_calling \
+ --model gpt-4o
+```
+
+Required environment variable:
+
+```bash
+export OPENAI_API_KEY=sk-...
+```
+
+### Anthropic
+
+```bash
+exgentic evaluate --benchmark gsm8k --agent tool_calling \
+ --model claude-3-5-sonnet-20241022
+```
+
+Required environment variable:
+
+```bash
+export ANTHROPIC_API_KEY=sk-ant-...
+```
+
+### Azure OpenAI
+
+```bash
+exgentic evaluate --benchmark gsm8k --agent tool_calling \
+ --model azure/
+```
+
+Required environment variables:
+
+```bash
+export AZURE_API_KEY=...
+export AZURE_API_BASE=https://.openai.azure.com
+export AZURE_API_VERSION=2024-02-01 # or whichever version your deployment uses
+```
+
+### AWS Bedrock
+
+```bash
+exgentic evaluate --benchmark gsm8k --agent tool_calling \
+ --model bedrock/anthropic.claude-3-5-sonnet-20241022-v2:0
+```
+
+Required environment variables:
+
+```bash
+export AWS_ACCESS_KEY_ID=...
+export AWS_SECRET_ACCESS_KEY=...
+export AWS_REGION_NAME=us-east-1
+```
+
+### Google Vertex AI
+
+```bash
+exgentic evaluate --benchmark gsm8k --agent tool_calling \
+ --model vertex_ai/gemini-1.5-pro
+```
+
+Required environment variables:
+
+```bash
+export VERTEXAI_PROJECT=my-gcp-project
+export VERTEXAI_LOCATION=us-central1
+```
+
+### Ollama (local)
+
+```bash
+exgentic evaluate --benchmark gsm8k --agent tool_calling \
+ --model ollama/llama3
+```
+
+Required: Ollama running locally. Set the base URL if it differs from the default:
+
+```bash
+export OLLAMA_API_BASE=http://localhost:11434 # default; only needed if different
+```
+
+### Any OpenAI-compatible endpoint
+
+```bash
+exgentic evaluate --benchmark gsm8k --agent tool_calling \
+ --model openai/
+```
+
+Override the base URL:
+
+```bash
+export OPENAI_API_BASE=http://localhost:8080/v1
+export OPENAI_API_KEY=any-non-empty-string # required by the client even if unused
+```
+
+This works with vLLM, LM Studio, LocalAI, Together AI, Fireworks, Anyscale, and any other OpenAI-compatible server.
+
+### LiteLLM proxy
+
+If you run a [LiteLLM proxy server](https://docs.litellm.ai/docs/proxy/quick_start) in front of your models:
+
+```bash
+exgentic evaluate --benchmark gsm8k --agent tool_calling \
+ --model openai/
+```
+
+```bash
+export OPENAI_API_BASE=http://localhost:4000
+export OPENAI_API_KEY=
+```
+
+---
+
+## Sampling parameters
+
+Use `--set agent.model.*` to control sampling. These map to `ModelSettings` and are forwarded to LiteLLM on every completion call.
+
+| Flag | Type | Default | Description |
+|------|------|---------|-------------|
+| `--set agent.model.temperature` | float ≥ 0 | `1.0` | Sampling temperature |
+| `--set agent.model.top_p` | float 0–1 | `null` | Nucleus sampling |
+| `--set agent.model.max_tokens` | int ≥ 0 | `null` | Maximum tokens in response |
+| `--set agent.model.reasoning_effort` | string | `null` | Reasoning effort level (o1/o3 models) |
+| `--set agent.model.num_retries` | int ≥ 0 | `5` | Retries on transient errors |
+| `--set agent.model.retry_after` | float ≥ 0 | `0.5` | Initial retry delay in seconds |
+| `--set agent.model.retry_strategy` | string | `exponential_backoff` | `exponential_backoff` or `constant` |
+
+Example — lower temperature and capped output:
+
+```bash
+exgentic evaluate --benchmark tau2 --agent tool_calling \
+ --model gpt-4o \
+ --set benchmark.user_simulator_model="gpt-4o" \
+ --set agent.model.temperature=0.2 \
+ --set agent.model.max_tokens=2048
+```
+
+---
+
+## Python API
+
+```python
+from exgentic import evaluate
+from exgentic.core.types import ModelSettings
+
+results = evaluate(
+ benchmark="tau2",
+ agent="tool_calling",
+ subset="retail",
+ num_tasks=5,
+ model="azure/my-gpt-4o-deployment",
+ agent_kwargs={
+ "model_settings": ModelSettings(
+ temperature=0.2,
+ max_tokens=2048,
+ num_retries=3,
+ )
+ },
+ benchmark_kwargs={"user_simulator_model": "gpt-4o"},
+)
+```
+
+---
+
+## Reasoning models
+
+For models that support reasoning effort (OpenAI o1, o3, etc.):
+
+```bash
+exgentic evaluate --benchmark swebench --agent tool_calling \
+ --model o3 \
+ --set agent.model.reasoning_effort=high \
+ --set agent.model.max_tokens=32768
+```
+
+Note: temperature is typically fixed at 1 for reasoning models and will be ignored if set.
+
+---
+
+## Cost tracking
+
+Exgentic records token counts and estimated cost for every LiteLLM completion automatically. Results appear in:
+
+- `outputs//results.json` — aggregate cost across all sessions
+- `outputs//sessions//results.json` — per-session cost
+
+Cost estimates are calculated using LiteLLM's built-in pricing database. For providers or custom deployments not in the database, cost will show as `0`.
+
+---
+
+## Caching
+
+LiteLLM-level response caching is enabled by default. To disable it for a run:
+
+```bash
+export EXGENTIC_LITELLM_CACHING=false
+```
+
+The cache directory defaults to `.litellm_cache` in the working directory. To move it:
+
+```bash
+export EXGENTIC_LITELLM_CACHE_DIR=/path/to/cache
+```
+
+---
+
+## Observability
+
+All LLM inference calls emit OpenTelemetry spans automatically when tracing is enabled. See [Observability Quick Start](./observability/quickstart.md) to set up tracing, and [Semantic Conventions](./observability/semantic-conventions.md) for the full attribute reference.
+
+---
+
+## Further reading
+
+- [LiteLLM providers documentation](https://docs.litellm.ai/docs/providers)
+- [Adding a new agent adapter](./adding-agents.md) — relevant when wrapping a framework that manages its own LLM calls
+- [Observability Quick Start](./observability/quickstart.md)
diff --git a/labs/AgentStream/exgentic/docs/huggingface.md b/labs/AgentStream/exgentic/docs/huggingface.md
new file mode 100644
index 00000000..05ce9104
--- /dev/null
+++ b/labs/AgentStream/exgentic/docs/huggingface.md
@@ -0,0 +1,35 @@
+# HuggingFace
+
+## Using HuggingFace Models
+
+Set your HF token and use the `huggingface///` model string format:
+
+```bash
+export HF_TOKEN=hf_...
+```
+
+```bash
+exgentic evaluate \
+ --benchmark gsm8k \
+ --agent tool_calling \
+ --model huggingface/together/meta-llama/Llama-3.1-70B-Instruct
+```
+
+LiteLLM routes the call through HuggingFace's inference providers (billed to your HF account). Supported providers include `together`, `sambanova`, and others. Tool calling support depends on the provider and model.
+
+## Running on HuggingFace Jobs
+
+HuggingFace Jobs run containerized workloads on HF infrastructure (requires Pro/Team/Enterprise).
+
+```bash
+hf jobs run astral-sh/uv:python3.12-bookworm sh -c "
+ uvx exgentic evaluate \
+ --benchmark gsm8k \
+ --agent tool_calling \
+ --model huggingface/together/meta-llama/Llama-3.1-70B-Instruct \
+ --output-dir /tmp/outputs &&
+ uvx exgentic batch publish --repo-id your-org/eval-results /tmp/outputs
+" --env HF_TOKEN=hf_...
+```
+
+Results are published to `https://huggingface.co/datasets/your-org/eval-results`.
diff --git a/labs/AgentStream/exgentic/docs/observability/quickstart.md b/labs/AgentStream/exgentic/docs/observability/quickstart.md
new file mode 100644
index 00000000..cfe52cde
--- /dev/null
+++ b/labs/AgentStream/exgentic/docs/observability/quickstart.md
@@ -0,0 +1,151 @@
+# Observability Quick Start
+
+This guide gets you from zero to traces in five minutes using Jaeger as a local trace collector.
+
+For a full reference of every attribute Exgentic emits, see [Semantic Conventions](./semantic-conventions.md).
+
+---
+
+## Prerequisites
+
+- Docker or Podman installed and running
+- `exgentic` installed with the `otel` extra (see below)
+
+---
+
+## Step 1 — Install the OTEL extra
+
+```bash
+uv sync --extra otel
+```
+
+---
+
+## Step 2 — Start Jaeger
+
+```bash
+# Using Docker (or replace 'docker' with 'podman')
+docker run -d --name jaeger \
+ -e COLLECTOR_OTLP_ENABLED=true \
+ -p 16686:16686 \
+ -p 4317:4317 \
+ -p 4318:4318 \
+ jaegertracing/all-in-one:latest
+```
+
+Default ports:
+
+| Port | Service |
+|-------|--------------|
+| 16686 | Jaeger UI |
+| 4317 | OTLP gRPC |
+| 4318 | OTLP HTTP |
+
+---
+
+## Step 3 — Configure environment variables
+
+```bash
+export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
+export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf # or 'grpc' for port 4317
+export EXGENTIC_OTEL_ENABLED=true
+```
+
+To include task prompts, tool arguments, and LLM messages in traces (opt-in — may contain sensitive data):
+
+```bash
+export EXGENTIC_OTEL_RECORD_CONTENT=true
+```
+
+---
+
+## Step 4 — Set up and run an evaluation
+
+```bash
+exgentic install --agent tool_calling
+exgentic install --benchmark tau2
+
+exgentic evaluate \
+ --benchmark tau2 \
+ --agent tool_calling \
+ --model gpt-4o \
+ --set benchmark.user_simulator_model="gpt-4o" \
+ --task 1 \
+ --max-steps 10
+```
+
+---
+
+## Step 5 — View traces
+
+Open [http://localhost:16686](http://localhost:16686), select the `exgentic` service, and click **Find Traces**.
+
+---
+
+## Exporting traces
+
+### Via the Jaeger UI
+
+1. Open a trace.
+2. Click the **JSON** button in the top-right corner.
+
+### Via the API
+
+```bash
+# All recent traces
+curl "http://localhost:16686/api/traces?service=exgentic&limit=100" | jq '.' > traces.json
+
+# A specific trace
+curl "http://localhost:16686/api/traces/" | jq '.' > trace.json
+```
+
+---
+
+## Troubleshooting
+
+### No traces appearing
+
+1. Verify environment variables are set: `env | grep OTEL`
+2. Confirm Jaeger is running: `docker ps | grep jaeger`
+3. Check Jaeger logs: `docker logs jaeger`
+4. Ensure the evaluation completed successfully before looking for traces
+
+### Traces are incomplete
+
+- Wait a few seconds after the evaluation finishes — spans are flushed asynchronously
+- Check session logs for OTEL-related errors
+- Verify network connectivity to Jaeger
+
+### Jaeger not starting
+
+```bash
+# Check for an existing container
+docker ps -a | grep jaeger
+
+# Stop and remove, then restart
+docker stop jaeger && docker rm jaeger
+docker run -d --name jaeger \
+ -e COLLECTOR_OTLP_ENABLED=true \
+ -p 16686:16686 -p 4317:4317 -p 4318:4318 \
+ jaegertracing/all-in-one:latest
+```
+
+---
+
+## Cleanup
+
+```bash
+docker stop jaeger
+docker rm jaeger
+
+# Optional: remove the image
+docker rmi jaegertracing/all-in-one:latest
+```
+
+---
+
+## Further reading
+
+- [Semantic Conventions](./semantic-conventions.md) — full attribute reference
+- [Jaeger documentation](https://www.jaegertracing.io/docs/)
+- [OpenTelemetry GenAI conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)
diff --git a/labs/AgentStream/exgentic/docs/observability/semantic-conventions.md b/labs/AgentStream/exgentic/docs/observability/semantic-conventions.md
new file mode 100644
index 00000000..92384e25
--- /dev/null
+++ b/labs/AgentStream/exgentic/docs/observability/semantic-conventions.md
@@ -0,0 +1,172 @@
+# Semantic Conventions
+
+This document maps Exgentic's core types to [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/). It reflects the actual implementation in `src/exgentic/observers/handlers/otel.py` and `src/exgentic/integrations/litellm/trace_logger.py`.
+
+For setup instructions, see [Quick Start](./quickstart.md).
+
+---
+
+## Span hierarchy
+
+```
+Session Span (ROOT)
+├── execute_tool initial_observation
+├── chat {model} ← LLM inference
+├── execute_tool {tool_name}
+├── chat {model} ← LLM inference
+├── execute_tool {tool_name}
+└── ... ← continues until session ends
+```
+
+---
+
+## Attribute reference
+
+The table below documents every attribute actually emitted by the implementation, organised by span type.
+
+| Span type | OTel attribute | Exgentic source | Type | Requirement | Content-filtered | Notes |
+|-----------|---------------|-----------------|------|-------------|-----------------|-------|
+| **Session (ROOT)** | `exgentic.benchmark.slug_name` | `BenchmarkEntry.slug_name` | string | Custom | No | Heritable |
+| **Session (ROOT)** | `exgentic.benchmark.subset` | `RunConfig.subset` | string | Custom | No | Heritable |
+| **Session (ROOT)** | `exgentic.benchmark.agent.name` | `AgentEntry.display_name` | string | Custom | No | Heritable |
+| **Session (ROOT)** | `exgentic.agent.slug` | `RunConfig.agent` | string | Custom | No | Heritable |
+| **Session (ROOT)** | `exgentic.run.id` | `Context.run_id` | string | Custom | No | Heritable |
+| **Session (ROOT)** | `gen_ai.request.model` | `RunConfig.model` | string | Recommended | No | Heritable; set when model is known at run start |
+| **Session (ROOT)** | `gen_ai.conversation.id` | `Session.session_id` | string | Recommended | No | Heritable; primary correlation attribute |
+| **Session (ROOT)** | `exgentic.session.id` | `Session.session_id` | string | Custom | No | Heritable; kept for backwards compatibility |
+| **Session (ROOT)** | `exgentic.session.task_id` | `Session.task_id` | string | Custom | No | |
+| **Session (ROOT)** | `exgentic.session.task` | `Session.task` | string | Opt-in | **Yes** | Task prompt; requires `EXGENTIC_OTEL_RECORD_CONTENT=true` |
+| **Session (ROOT)** | `exgentic.session.action.{name}.name` | `ActionType.name` | string | Custom | No | One entry per action in `Session.actions` |
+| **Session (ROOT)** | `exgentic.session.action.{name}.description` | `ActionType.description` | string | Custom | No | |
+| **Session (ROOT)** | `exgentic.session.action.{name}.is_message` | `ActionType.is_message` | bool | Custom | No | |
+| **Session (ROOT)** | `exgentic.session.action.{name}.is_finish` | `ActionType.is_finish` | bool | Custom | No | |
+| **Session (ROOT)** | `exgentic.context.{key}` | `Session.context[key]` | string | Custom | No | One entry per context key |
+| **Session (ROOT)** | `exgentic.session.agent.id` | `AgentInstance.agent_id` | string | Custom | No | |
+| **Session (ROOT)** | `exgentic.session.agent.path` | `AgentInstance.paths.agent_dir` | string | Custom | No | |
+| **Session (ROOT)** | `exgentic.score.success` | `SessionScore.success` | bool | Custom | No | Set on session close |
+| **Session (ROOT)** | `exgentic.score` | `SessionScore.score` | float | Custom | No | Set on session close |
+| **Session (ROOT)** | `exgentic.score.is_finished` | `SessionScore.is_finished` | bool | Custom | No | Set on session close |
+| **Session (ROOT)** | `exgentic.session.steps` | step counter | int | Custom | No | Set on session close |
+| **Session (ROOT)** | `exgentic.agent.agent_cost` | `AgentInstance.get_cost()` | string (JSON) | Custom | No | Set on session close |
+| **Session (ROOT)** | `exgentic.session.cost` | `Session.get_cost()` | string (JSON) | Custom | No | Set on session close |
+| **execute_tool** | `gen_ai.operation.name` | `"execute_tool"` | string | Required | No | Constant value |
+| **execute_tool** | `gen_ai.tool.name` | `Action.name` | string | Required | No | |
+| **execute_tool** | `gen_ai.tool.id` | `Action.id` | string | Recommended | No | |
+| **execute_tool** | `gen_ai.tool.description` | `ActionType.description` | string | Recommended | No | Looked up from `Session.actions` |
+| **execute_tool** | `gen_ai.tool.parameters` | `Action.arguments` | string (JSON) | Opt-in | **Yes** | Requires `EXGENTIC_OTEL_RECORD_CONTENT=true` |
+| **execute_tool** | `gen_ai.tool.result` | `Observation` | string | Opt-in | **Yes** | Requires `EXGENTIC_OTEL_RECORD_CONTENT=true` |
+| **execute_tool** | `gen_ai.conversation.id` | `Session.session_id` | string | Recommended | No | Inherited from session span |
+| **LLM inference** | `gen_ai.operation.name` | `"chat"` or `"text_completion"` | string | Required | No | |
+| **LLM inference** | `gen_ai.provider.name` | `litellm_params.custom_llm_provider` | string | Required | No | Mapped to standard provider names |
+| **LLM inference** | `gen_ai.request.model` | `LitellmKwargs.model` | string | Required | No | |
+| **LLM inference** | `error.type` | exception class name | string | Required | No | Set on failure |
+| **LLM inference** | `gen_ai.conversation.id` | `Context.session_id` | string | Recommended | No | |
+| **LLM inference** | `gen_ai.request.max_tokens` | `optional_params.max_tokens` | int | Recommended | No | |
+| **LLM inference** | `gen_ai.request.temperature` | `optional_params.temperature` | float | Recommended | No | |
+| **LLM inference** | `gen_ai.request.top_p` | `optional_params.top_p` | float | Recommended | No | |
+| **LLM inference** | `gen_ai.request.top_k` | `optional_params.top_k` | float | Recommended | No | |
+| **LLM inference** | `gen_ai.request.frequency_penalty` | `optional_params.frequency_penalty` | float | Recommended | No | |
+| **LLM inference** | `gen_ai.request.presence_penalty` | `optional_params.presence_penalty` | float | Recommended | No | |
+| **LLM inference** | `gen_ai.request.stop_sequences` | `optional_params.stop` | string[] | Recommended | No | |
+| **LLM inference** | `gen_ai.request.choice.count` | `optional_params.n` | int | Required | No | Only when `n != 1` |
+| **LLM inference** | `gen_ai.request.seed` | `optional_params.seed` | int | Required | No | |
+| **LLM inference** | `gen_ai.response.id` | `ResponseObject.id` | string | Recommended | No | |
+| **LLM inference** | `gen_ai.response.model` | `ResponseObject.model` | string | Recommended | No | Actual model resolved by the provider |
+| **LLM inference** | `gen_ai.usage.input_tokens` | `usage.prompt_tokens` | int | Recommended | No | |
+| **LLM inference** | `gen_ai.usage.output_tokens` | `usage.completion_tokens` | int | Recommended | No | |
+| **LLM inference** | `gen_ai.response.finish_reasons` | `choices[*].finish_reason` | string[] | Recommended | No | |
+| **LLM inference** | `gen_ai.tool.definitions` | `LitellmKwargs.tools` | string (JSON) | Opt-in | **Yes** | Requires `EXGENTIC_OTEL_RECORD_CONTENT=true` |
+| **LLM inference** | `gen_ai.input.messages` | `LitellmKwargs.messages` | string (JSON) | Opt-in | **Yes** | Requires `EXGENTIC_OTEL_RECORD_CONTENT=true` |
+| **LLM inference** | `gen_ai.output.messages` | `choices[*].message` | string (JSON) | Opt-in | **Yes** | Requires `EXGENTIC_OTEL_RECORD_CONTENT=true` |
+
+---
+
+## Span details
+
+### Session span (ROOT)
+
+- **Name**: `{benchmark_name} {subset} session`
+- **Kind**: `INTERNAL`
+- **Opened**: `OtelTracingObserver.on_session_creation`
+- **Closed**: `OtelTracingObserver.on_session_success` or `on_session_error`
+
+### execute_tool span
+
+- **Name**: `execute_tool {tool_name}` or `execute_tool initial_observation`
+- **Kind**: `CLIENT`
+- **Opened**: `OtelTracingObserver.on_session_start` (initial), `on_react_success`, or `on_react_error`
+- **Closed**: `OtelTracingObserver.on_step_success` or `on_step_error`
+- **Reference**: [OTel GenAI execute_tool span](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/#execute-tool-span)
+
+### LLM inference span
+
+- **Name**: `{operation} {model}` (e.g., `chat gpt-4o`)
+- **Kind**: `CLIENT`
+- **Opened/Closed**: `TraceLogger._write_otel` (LiteLLM callback)
+- **Parent**: session span (via OTEL context propagation)
+- **Reference**: [OTel GenAI inference span](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/#inference)
+
+---
+
+## Attribute inheritance
+
+The following attributes are set on the session span and automatically propagated to all child spans via `SessionSpanManager.set_heritable_attribute()`:
+
+| Attribute | Source |
+|-----------|--------|
+| `gen_ai.conversation.id` | `Session.session_id` — primary correlation key |
+| `exgentic.session.id` | `Session.session_id` — backwards compatibility |
+| `gen_ai.request.model` | `RunConfig.model` (when available) |
+| `exgentic.run.id` | `Context.run_id` |
+| `exgentic.benchmark.slug_name` | `BenchmarkEntry.slug_name` |
+| `exgentic.benchmark.subset` | `RunConfig.subset` |
+| `exgentic.benchmark.agent.name` | `AgentEntry.display_name` |
+| `exgentic.agent.slug` | `RunConfig.agent` |
+
+---
+
+## Content filtering
+
+Attributes marked **Yes** in the content-filtered column contain user data (prompts, tool arguments, model responses). They are **not recorded by default** and must be explicitly enabled:
+
+```bash
+export EXGENTIC_OTEL_RECORD_CONTENT=true
+```
+
+Attributes that are never filtered include all IDs, names, counters, scores, and static schemas — only runtime user content requires opt-in.
+
+---
+
+## Implementation notes
+
+### Model name resolution
+
+Because `AgentInstance` does not expose model settings, the model name is extracted from `RunConfig` at run start:
+
+```python
+model_name = run_config.model or (run_config.agent_kwargs or {}).get("model")
+```
+
+### Cost attributes
+
+`LiteLLMCostReport` and `UpdatableCostReport` are serialized to JSON strings for OTEL compatibility:
+
+- `exgentic.agent.agent_cost` — agent-level cost report
+- `exgentic.session.cost` — full session cost report
+
+### LLM span parent context
+
+LLM inference spans are created inside the LiteLLM callback and attached to the session span via OTEL context propagation:
+
+1. The session span manager writes the current OTEL context into the `Context` ContextVar via `update_tracing_context()`.
+2. The LiteLLM trace logger reads the OTEL context from that ContextVar.
+3. LLM spans are created with the session span as their parent using `_get_parent_context()`.
+
+---
+
+## References
+
+- [OTel GenAI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)
+- [execute_tool span spec](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/#execute-tool-span)
+- [Inference span spec](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/#inference)
+- [Quick Start](./quickstart.md)
diff --git a/labs/AgentStream/exgentic/docs/observers.md b/labs/AgentStream/exgentic/docs/observers.md
new file mode 100644
index 00000000..bd29313b
--- /dev/null
+++ b/labs/AgentStream/exgentic/docs/observers.md
@@ -0,0 +1,221 @@
+# Observers and Controllers
+
+Observers let you hook into the evaluation lifecycle to add custom logging, monitoring, alerting, or analytics — without modifying benchmarks or agents. Controllers extend this with the ability to stop a run early.
+
+**Related docs:**
+[docs/](./README.md) · [Python API](./python-api.md) · [Output Format](./output-format.md) · [Observability](./observability/quickstart.md)
+
+---
+
+## Observer interface
+
+All observers extend `exgentic.core.orchestrator.observer.Observer`. Every method has a default no-op implementation, so you only override what you need.
+
+```python
+from exgentic.core.orchestrator.observer import Observer
+
+class Observer:
+ # Run-level callbacks
+ def on_run_start(self, run_config) -> None: ...
+ def on_run_success(self, results, run_config) -> None: ...
+ def on_run_error(self, error) -> None: ...
+
+ # Session-level callbacks
+ def on_session_creation(self, session) -> None: ...
+ def on_session_start(self, session, agent, observation) -> None: ...
+ def on_session_scoring(self, session) -> None: ...
+ def on_session_success(self, session, score, agent) -> None: ...
+ def on_session_error(self, session, error) -> None: ...
+ def on_session_reuse(self, task_result) -> None: ...
+
+ # Step-level: agent.react() returned an action
+ def on_react_success(self, session, action) -> None: ...
+ def on_react_error(self, session, error) -> None: ...
+
+ # Step-level: session.step(action) returned an observation
+ def on_step_success(self, session, observation) -> None: ...
+ def on_step_error(self, session, error) -> None: ...
+```
+
+### Callback parameters
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `run_config` | `RunConfig` | The run configuration |
+| `results` | `RunResults` | Aggregated results (available in `on_run_success`) |
+| `session` | `Session` | Session object with `session_id`, `task_id`, `paths` |
+| `agent` | `Agent` | Agent config object with `get_cost()` |
+| `observation` | `Observation \| None` | Observation returned by the benchmark (None on first step) |
+| `action` | `Action \| None` | Action returned by the agent (None if agent is done) |
+| `score` | `SessionScore` | Score with `score`, `success`, `is_finished`, `session_metrics`, `session_metadata` |
+| `task_result` | `SessionResults` | Results for a session that was skipped/reused from cache |
+| `error` | `Exception` | The exception that occurred |
+
+---
+
+## Using observers
+
+Pass observers to `evaluate()`, `execute()`, or `aggregate()`:
+
+```python
+from exgentic import evaluate
+
+results = evaluate(
+ benchmark="tau2",
+ agent="tool_calling",
+ num_tasks=10,
+ observers=[MyObserver()],
+)
+```
+
+Multiple observers are supported:
+
+```python
+observers=[LoggingObserver(), MetricsObserver(), AlertObserver()]
+```
+
+---
+
+## Examples
+
+### Print session results as they complete
+
+```python
+from exgentic.core.orchestrator.observer import Observer
+
+class PrintObserver(Observer):
+ def on_session_success(self, session, score, agent):
+ status = "PASS" if score.success else "FAIL"
+ print(f"[{status}] {session.task_id} score={score.score:.2f} cost=${agent.get_cost().total_cost:.4f}")
+
+ def on_session_error(self, session, error):
+ print(f"[ERROR] {session.task_id} {type(error).__name__}: {error}")
+```
+
+### Track costs in real time
+
+```python
+from exgentic.core.orchestrator.observer import Observer
+
+class CostTracker(Observer):
+ def __init__(self):
+ self.total_cost = 0.0
+
+ def on_session_success(self, session, score, agent):
+ self.total_cost += agent.get_cost().total_cost
+ print(f"Running total: ${self.total_cost:.4f}")
+```
+
+### Write a custom log file
+
+```python
+import json
+from pathlib import Path
+from exgentic.core.orchestrator.observer import Observer
+
+class JsonlLogger(Observer):
+ def __init__(self, path: str):
+ self.path = Path(path)
+
+ def on_session_success(self, session, score, agent):
+ entry = {
+ "session_id": session.session_id,
+ "task_id": session.task_id,
+ "success": score.success,
+ "score": score.score,
+ }
+ with self.path.open("a") as f:
+ f.write(json.dumps(entry) + "\n")
+```
+
+### Send a Slack alert on failure
+
+```python
+import requests
+from exgentic.core.orchestrator.observer import Observer
+
+class SlackAlerter(Observer):
+ def __init__(self, webhook_url: str):
+ self.webhook_url = webhook_url
+
+ def on_session_error(self, session, error):
+ requests.post(self.webhook_url, json={
+ "text": f":x: Session `{session.task_id}` failed: `{error}`"
+ })
+```
+
+---
+
+## Controllers
+
+Controllers extend observers with the ability to raise errors that stop the run. Use them to implement early stopping — e.g. abort if too many consecutive failures occur.
+
+```python
+from exgentic.core.orchestrator.observer import Observer
+
+class EarlyStopController(Observer):
+ def __init__(self, max_failures: int = 3):
+ self.max_failures = max_failures
+ self.failures = 0
+
+ def on_session_error(self, session, error):
+ self.failures += 1
+ if self.failures >= self.max_failures:
+ raise RuntimeError(f"Stopping: {self.failures} consecutive session errors")
+```
+
+Pass controllers via the `controllers` parameter:
+
+```python
+results = evaluate(
+ benchmark="tau2",
+ agent="tool_calling",
+ num_tasks=50,
+ controllers=[EarlyStopController(max_failures=5)],
+)
+```
+
+---
+
+## Built-in observers
+
+Exgentic uses these observers internally. They run automatically — you do not need to register them.
+
+| Observer | What it does |
+|----------|-------------|
+| `ResultsObserver` | Writes `trajectory.jsonl` and `results.json` for every session; computes `RunResults` |
+| `LoggerObserver` | Logs run progress to the console |
+| `FileLoggerObserver` | Writes `run.log` and per-session logs |
+| `OtelTracingObserver` | Emits OpenTelemetry spans (active when `EXGENTIC_OTEL_ENABLED=true`) |
+| `DashboardEventsObserver` | Streams events to the live dashboard |
+| `WarningsObserver` | Captures and writes `warnings.log` |
+| `RecapObserver` | Prints a summary table at the end of a run |
+
+---
+
+## Thread safety
+
+Observers may be called from multiple threads when `max_workers > 1`. If your observer maintains shared state (counters, file handles, accumulators), protect it with a lock:
+
+```python
+import threading
+from exgentic.core.orchestrator.observer import Observer
+
+class ThreadSafeCounter(Observer):
+ def __init__(self):
+ self._lock = threading.Lock()
+ self.count = 0
+
+ def on_session_success(self, session, score, agent):
+ with self._lock:
+ self.count += 1
+```
+
+---
+
+## See also
+
+- [Python API](./python-api.md) — how to pass observers to `evaluate()`
+- [Observability Quick Start](./observability/quickstart.md) — OpenTelemetry tracing (built-in observer)
+- [Output Format](./output-format.md) — the data that `ResultsObserver` writes
+- [docs/](./README.md) — documentation index
diff --git a/labs/AgentStream/exgentic/docs/output-format.md b/labs/AgentStream/exgentic/docs/output-format.md
new file mode 100644
index 00000000..8d7e9eb7
--- /dev/null
+++ b/labs/AgentStream/exgentic/docs/output-format.md
@@ -0,0 +1,325 @@
+# Output Format
+
+Every evaluation writes structured results to a directory under `outputs//`. This document covers the file layout and the schema of every results file.
+
+**Related docs:**
+[docs/](./README.md) · [Python API](./python-api.md) · [Batch Runs](./batch.md) · [CLI Reference](./cli-reference.md)
+
+---
+
+## Directory layout
+
+```
+outputs//
+├── results.json # Aggregated run-level results
+├── benchmark_results.json # Benchmark-specific aggregated results
+├── run/
+│ ├── config.json # Snapshot of RunConfig used for this run
+│ ├── run.log # Main execution log
+│ └── warnings.log # Warnings captured during execution
+└── sessions//
+ ├── config.json # SessionConfig for this task
+ ├── results.json # Session-level results
+ ├── trajectory.jsonl # One JSON line per step (action + observation)
+ ├── agent/
+ │ └── agent.log # Agent execution log
+ └── benchmark/
+ ├── results.json # Benchmark-specific session results
+ └── session.log # Benchmark session log
+```
+
+### run_id and session_id
+
+Both IDs are deterministic SHA256 hashes of the configuration:
+
+- `run_id` — 12-character prefix of the hash of `(benchmark, agent, subset, model, benchmark_kwargs, agent_kwargs)` — everything except the task list.
+- `session_id` — 8-character prefix of the hash of `(benchmark, agent, subset, task_id, model, benchmark_kwargs, agent_kwargs)` — includes the task, so each task gets a stable ID.
+
+This means re-running the same config produces the same IDs. Completed sessions are skipped unless `overwrite_sessions` is set.
+
+---
+
+## results.json
+
+The top-level aggregated results for the entire run.
+
+```json
+{
+ "benchmark_name": "Tau2Bench",
+ "benchmark_slug_name": "tau2",
+ "agent_name": "LiteLLM Tool Calling",
+ "agent_slug_name": "tool_calling",
+ "model_name": "gpt-4o",
+ "model_names": ["gpt-4o"],
+ "subset_name": "retail",
+ "total_sessions": 10,
+ "planned_sessions": 10,
+ "successful_sessions": 7,
+
+ "benchmark_score": 0.72,
+ "average_score": 0.65,
+
+ "average_agent_cost": 0.043,
+ "total_agent_cost": 0.43,
+ "average_benchmark_cost": 0.021,
+ "total_benchmark_cost": 0.21,
+ "total_run_cost": 0.64,
+
+ "average_steps": 12.4,
+ "average_action_count": 15.1,
+ "average_invalid_action_count": 0.8,
+ "average_invalid_action_percent": 5.3,
+
+ "percent_finished": 80.0,
+ "percent_successful": 70.0,
+ "percent_finished_successful": 70.0,
+ "percent_finished_unsuccessful": 10.0,
+ "percent_unfinished": 10.0,
+ "percent_error": 10.0,
+
+ "aggregation_mode": "completed_only",
+ "completed_sessions": 10,
+ "incomplete_sessions": 0,
+ "missing_sessions": 0,
+ "aggregated_session_ids": ["a1b2c3d4", "..."],
+ "skipped_session_ids": [],
+ "skipped_session_reasons": {},
+
+ "exgentic_version": "0.3.0",
+
+ "session_results": [ ... ]
+}
+```
+
+### Field reference
+
+#### Identity
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `benchmark_name` | string | Benchmark display name |
+| `benchmark_slug_name` | string | Benchmark CLI identifier |
+| `agent_name` | string | Agent display name |
+| `agent_slug_name` | string | Agent CLI identifier |
+| `model_name` | string \| null | Primary model used |
+| `model_names` | list[string] \| null | All models used (if multiple) |
+| `subset_name` | string \| null | Benchmark subset |
+| `exgentic_version` | string \| null | Exgentic version that produced these results |
+
+#### Session counts
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `total_sessions` | int | Sessions that were executed |
+| `planned_sessions` | int \| null | Sessions originally planned (from `num_tasks` or `task_ids`) |
+| `successful_sessions` | int | Sessions where `success=true` |
+
+#### Scores
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `benchmark_score` | float \| null | Primary score from `Benchmark.aggregate_sessions()` — benchmark-specific |
+| `average_score` | float \| null | Mean of per-session `score` values |
+
+#### Costs
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `average_agent_cost` | float \| null | Mean agent API cost per session (USD) |
+| `total_agent_cost` | float \| null | Total agent API cost across all sessions |
+| `average_benchmark_cost` | float \| null | Mean benchmark API cost per session (e.g. simulator LLM) |
+| `total_benchmark_cost` | float \| null | Total benchmark API cost |
+| `total_run_cost` | float \| null | Total cost (agent + benchmark) |
+
+#### Performance statistics
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `average_steps` | float \| null | Mean number of steps per session |
+| `average_action_count` | float \| null | Mean number of actions per session |
+| `average_invalid_action_count` | float \| null | Mean invalid actions per session |
+| `average_invalid_action_percent` | float \| null | Invalid actions as percentage of total |
+
+#### Outcome breakdown
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `percent_finished` | float \| null | Sessions that reached a terminal state (success or failure) |
+| `percent_successful` | float \| null | Sessions with `success=true` |
+| `percent_finished_successful` | float \| null | Sessions that finished successfully |
+| `percent_finished_unsuccessful` | float \| null | Sessions that finished unsuccessfully |
+| `percent_unfinished` | float \| null | Sessions that ran out of steps |
+| `percent_error` | float \| null | Sessions that raised an exception |
+
+#### Aggregation provenance
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `aggregation_mode` | string \| null | Always `"completed_only"` — only completed sessions are aggregated |
+| `completed_sessions` | int \| null | Sessions with a `results.json` on disk |
+| `incomplete_sessions` | int \| null | Sessions with a directory but no `results.json` |
+| `missing_sessions` | int \| null | Planned sessions with no directory at all |
+| `aggregated_session_ids` | list[string] \| null | Sessions included in score aggregation |
+| `skipped_session_ids` | list[string] \| null | Sessions excluded from aggregation |
+| `skipped_session_reasons` | dict \| null | Reason per skipped session ID |
+
+---
+
+## sessions//results.json
+
+Per-session results.
+
+```json
+{
+ "session_id": "a1b2c3d4",
+ "task_id": "retail_1",
+ "success": true,
+ "score": 1.0,
+ "is_finished": true,
+ "status": "success",
+ "steps": 14,
+ "action_count": 17,
+ "invalid_action_count": 1,
+ "agent_cost": 0.038,
+ "benchmark_cost": 0.019,
+ "execution_time": 42.3,
+ "details": { ... },
+ "cost_reports": {
+ "agent": { "model_name": "gpt-4o", "input_tokens": 9400, "output_tokens": 820, ... },
+ "benchmark": { ... }
+ }
+}
+```
+
+### Field reference
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `session_id` | string | 8-char deterministic session ID |
+| `task_id` | string \| null | Task identifier from the benchmark |
+| `success` | bool | Whether the session ended successfully |
+| `score` | float \| null | Benchmark-assigned score for this session (0–1 unless benchmark uses a different scale) |
+| `is_finished` | bool \| null | Whether the agent signalled completion (as opposed to hitting a step limit) |
+| `status` | string | Session outcome status (see below) |
+| `steps` | int | Number of (action → observation) steps executed |
+| `action_count` | int | Total individual actions taken |
+| `invalid_action_count` | int | Actions that failed schema or contract validation |
+| `agent_cost` | float | Estimated agent API cost (USD) |
+| `benchmark_cost` | float | Estimated benchmark API cost (USD) |
+| `execution_time` | float | Wall-clock time in seconds |
+| `details` | object | Full `SessionScore` dump — benchmark-specific |
+| `cost_reports` | object | Detailed cost breakdown keyed by `"agent"` and `"benchmark"` |
+
+### Session outcome status
+
+| Value | Meaning |
+|-------|---------|
+| `success` | Session finished and benchmark scored it as successful |
+| `unsuccessful` | Session finished but benchmark scored it as unsuccessful |
+| `unfinished` | Agent never returned `None` — ran out of steps |
+| `limit_reached` | Hit `max_steps` or `max_actions` |
+| `error` | An exception occurred during execution |
+| `cancelled` | Run was cancelled before this session completed |
+| `unknown` | Status could not be determined |
+
+---
+
+## sessions//trajectory.jsonl
+
+A newline-delimited JSON file with one entry per step. Use this to replay or audit what the agent did.
+
+```json
+{"event": "observation", "step": 0, "initial": true, "session_id": "a1b2c3d4", "task_id": "retail_1", "observation": {...}, "action": null}
+{"event": "action", "step": 1, "initial": false, "session_id": "a1b2c3d4", "task_id": "retail_1", "observation": null, "action": {"name": "search_products", "arguments": {...}}}
+{"event": "observation", "step": 1, "initial": false, "session_id": "a1b2c3d4", "task_id": "retail_1", "observation": {"content": [...]}, "action": null}
+...
+```
+
+---
+
+## benchmark_results.json
+
+Benchmark-specific aggregated results produced by `Benchmark.aggregate_sessions()`. Schema is benchmark-defined, but always includes at minimum:
+
+```json
+{
+ "benchmark_name": "Tau2Bench",
+ "total_tasks": 10,
+ "score": 0.72,
+ "metrics": { ... }
+}
+```
+
+---
+
+## cost_reports schema
+
+Within session `results.json`, the `cost_reports` dict contains detailed token and cost breakdowns.
+
+```json
+{
+ "agent": {
+ "model_name": "gpt-4o",
+ "input_tokens": 9400,
+ "output_tokens": 820,
+ "input_cost": 0.0235,
+ "output_cost": 0.0164,
+ "total_cost": 0.0399
+ },
+ "benchmark": {
+ "model_name": "gpt-4o",
+ "input_tokens": 4200,
+ "output_tokens": 340,
+ "input_cost": 0.0105,
+ "output_cost": 0.0068,
+ "total_cost": 0.0173
+ }
+}
+```
+
+Cost estimates come from LiteLLM's pricing database. For providers or deployments not in the database, costs show as `0`.
+
+---
+
+## Reading results programmatically
+
+```python
+import json
+from pathlib import Path
+
+run_dir = Path("outputs/abc123def456")
+
+# Load run-level results
+results = json.loads((run_dir / "results.json").read_text())
+print(f"Score: {results['benchmark_score']}")
+print(f"Sessions: {results['total_sessions']}")
+
+# Load a specific session trajectory
+session_dir = run_dir / "sessions" / "a1b2c3d4"
+trajectory = [
+ json.loads(line)
+ for line in (session_dir / "trajectory.jsonl").read_text().splitlines()
+]
+```
+
+Or use the Python API to load and validate:
+
+```python
+from exgentic import results
+from exgentic.batch import RunConfig
+
+config = RunConfig(benchmark="tau2", agent="tool_calling", subset="retail")
+run_results = results(config)
+print(run_results.benchmark_score)
+```
+
+See [Python API](./python-api.md) for the full API reference.
+
+---
+
+## See also
+
+- [Python API](./python-api.md) — `results()`, `status()`, `aggregate()` functions
+- [Batch Runs](./batch.md) — `batch extract` to export results to CSV
+- [CLI Reference](./cli-reference.md) — `exgentic results` command
+- [docs/](./README.md) — documentation index
diff --git a/labs/AgentStream/exgentic/docs/python-api.md b/labs/AgentStream/exgentic/docs/python-api.md
new file mode 100644
index 00000000..a8dc49b4
--- /dev/null
+++ b/labs/AgentStream/exgentic/docs/python-api.md
@@ -0,0 +1,361 @@
+# Python API
+
+Exgentic can be used as a library. The public API is importable directly from the `exgentic` package.
+
+**Related docs:**
+[docs/](./README.md) · [CLI Reference](./cli-reference.md) · [Output Format](./output-format.md) · [Batch Runs](./batch.md) · [Custom Models](./custom-models.md)
+
+---
+
+## Installation
+
+```bash
+uv add exgentic # or: pip install exgentic
+```
+
+---
+
+## Quick example
+
+```python
+from exgentic import evaluate
+
+results = evaluate(
+ benchmark="tau2",
+ agent="tool_calling",
+ subset="retail",
+ num_tasks=5,
+ model="gpt-4o",
+ benchmark_kwargs={"user_simulator_model": "gpt-4o"},
+)
+
+print(results.benchmark_score)
+print(results.total_agent_cost)
+```
+
+---
+
+## Core functions
+
+All functions share the same config parameters. You can pass them as keyword arguments or as a pre-built `RunConfig` object.
+
+### evaluate()
+
+Run sessions and aggregate results. The standard function for most use cases.
+
+```python
+from exgentic import evaluate
+
+results = evaluate(
+ benchmark="tau2",
+ agent="tool_calling",
+ subset="retail",
+ num_tasks=10,
+ model="gpt-4o",
+ benchmark_kwargs={"user_simulator_model": "gpt-4o"},
+ agent_kwargs={"model_settings": {"temperature": 0.2}},
+ max_steps=100,
+ max_actions=100,
+ max_workers=4,
+)
+```
+
+Returns: `RunResults` — see [Output Format](./output-format.md) for the full schema.
+
+### execute()
+
+Run sessions without aggregating results. Use this when you want to separate execution from aggregation (e.g. run on multiple machines, aggregate centrally).
+
+```python
+from exgentic import execute, aggregate
+
+execute(benchmark="tau2", agent="tool_calling", subset="retail", num_tasks=10)
+# ... copy outputs to central machine ...
+results = aggregate(benchmark="tau2", agent="tool_calling", subset="retail", num_tasks=10)
+```
+
+Returns: `RunResults` with aggregation fields empty.
+
+### aggregate()
+
+Aggregate already-completed sessions without running anything. Reads `results.json` from each session directory and computes run-level statistics.
+
+```python
+from exgentic import aggregate
+
+results = aggregate(
+ benchmark="tau2",
+ agent="tool_calling",
+ subset="retail",
+ num_tasks=10,
+)
+```
+
+Returns: `RunResults`.
+
+### status()
+
+Get the current execution status of a run without running anything.
+
+```python
+from exgentic import status
+
+run_status = status(benchmark="tau2", agent="tool_calling", subset="retail", num_tasks=10)
+print(run_status.completed) # number of completed sessions
+print(run_status.running) # number of currently-running sessions
+print(run_status.missing) # number of not-yet-started sessions
+```
+
+Returns: `RunStatus`.
+
+### preview()
+
+Get the execution plan for a run — which sessions would run, which would be reused, etc. — without executing.
+
+```python
+from exgentic import preview
+from exgentic.interfaces.lib.api import RunConfig
+
+config = RunConfig(benchmark="tau2", agent="tool_calling", subset="retail", num_tasks=10)
+plan = preview(config)
+
+print(plan.to_run) # list of session configs that would run
+print(plan.reuse) # list of already-completed sessions
+print(plan.missing) # list of sessions with no output directory
+```
+
+Returns: `RunPlan`.
+
+### results()
+
+Load aggregated results from a completed run's `results.json` on disk.
+
+```python
+from exgentic import results
+from exgentic.interfaces.lib.api import RunConfig
+
+config = RunConfig(benchmark="tau2", agent="tool_calling", subset="retail", num_tasks=10)
+run_results = results(config)
+```
+
+Returns: `RunResults`.
+
+---
+
+## Parameters
+
+All core functions accept the same parameters (as kwargs or as a `RunConfig`/`SessionConfig` object).
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `benchmark` | string \| Benchmark | required | Benchmark slug or instance |
+| `agent` | string \| Agent | required | Agent slug or instance |
+| `subset` | string | null | Benchmark subset |
+| `task_ids` | list[string] | null | Explicit task IDs |
+| `num_tasks` | int | null | Number of tasks to run |
+| `model` | string | null | Model override (forwarded to the agent) |
+| `output_dir` | string | `./outputs` | Results directory |
+| `cache_dir` | string | null | Cache directory |
+| `run_id` | string | auto | Deterministic run ID derived from config |
+| `max_steps` | int | 100 | Maximum steps per session |
+| `max_actions` | int | 100 | Maximum actions per session |
+| `max_workers` | int | null | Parallel session workers |
+| `overwrite_sessions` | bool | false | Re-run completed sessions |
+| `benchmark_kwargs` | dict | null | Extra kwargs for the benchmark constructor |
+| `agent_kwargs` | dict | null | Extra kwargs for the agent constructor |
+| `observers` | list | null | Custom observers (see [Observers](./observers.md)) |
+| `controllers` | list | null | Custom controllers (see [Observers](./observers.md)) |
+
+---
+
+## Discovery functions
+
+### list_benchmarks()
+
+```python
+from exgentic import list_benchmarks
+
+for b in list_benchmarks():
+ print(b["slug_name"], b["display_name"], b["installed"])
+```
+
+Returns: `list[dict]` with keys `slug_name`, `display_name`, `installed`, `installed_at`.
+
+### list_agents()
+
+```python
+from exgentic import list_agents
+
+for a in list_agents():
+ print(a["slug_name"], a["display_name"])
+```
+
+Returns: `list[dict]` with keys `slug_name`, `display_name`, `installed`, `installed_at`.
+
+### list_subsets()
+
+```python
+from exgentic import list_subsets
+
+subsets = list_subsets("tau2")
+# ["retail", "airline", "banking"]
+```
+
+Returns: `list[str]`.
+
+### list_tasks()
+
+```python
+from exgentic import list_tasks
+
+tasks = list_tasks(benchmark="tau2", subset="retail")
+# ["retail_1", "retail_2", ...]
+```
+
+Returns: `list[str]`.
+
+---
+
+## Setup functions
+
+### setup_benchmark()
+
+Install a benchmark's dependencies and run its `setup.sh`. Equivalent to `exgentic install --benchmark `.
+
+```python
+from exgentic.interfaces.lib.api import setup_benchmark
+
+setup_benchmark("tau2")
+setup_benchmark("tau2", force=True) # reinstall even if already set up
+setup_benchmark("tau2", runner="venv") # install into isolated venv
+```
+
+### setup_agent()
+
+```python
+from exgentic.interfaces.lib.api import setup_agent
+
+setup_agent("tool_calling")
+setup_agent("tool_calling", force=True)
+```
+
+---
+
+## Config objects
+
+Use config objects when you want to construct a run programmatically, save configs to disk, or pass them around.
+
+### RunConfig
+
+```python
+from exgentic.interfaces.lib.api import RunConfig
+
+config = RunConfig(
+ benchmark="tau2",
+ agent="tool_calling",
+ subset="retail",
+ num_tasks=10,
+ model="gpt-4o",
+ max_steps=100,
+ benchmark_kwargs={"user_simulator_model": "gpt-4o"},
+ agent_kwargs={"model_settings": {"temperature": 0.2}},
+)
+
+# Save to disk
+import json
+Path("my_run.json").write_text(config.model_dump_json(indent=2))
+
+# Load from disk
+config2 = RunConfig.model_validate_json(Path("my_run.json").read_text())
+```
+
+### SessionConfig
+
+For single-task runs:
+
+```python
+from exgentic.interfaces.lib.api import SessionConfig
+
+config = SessionConfig(
+ benchmark="tau2",
+ agent="tool_calling",
+ task_id="retail_1",
+ subset="retail",
+ model="gpt-4o",
+)
+```
+
+---
+
+## Model settings
+
+Pass model settings through `agent_kwargs`:
+
+```python
+from exgentic import evaluate
+from exgentic.core.types import ModelSettings
+
+results = evaluate(
+ benchmark="tau2",
+ agent="tool_calling",
+ subset="retail",
+ num_tasks=5,
+ model="gpt-4o",
+ agent_kwargs={
+ "model_settings": ModelSettings(
+ temperature=0.2,
+ max_tokens=4096,
+ num_retries=3,
+ )
+ },
+)
+```
+
+Or as a plain dict (equivalent):
+
+```python
+agent_kwargs={
+ "model_settings": {
+ "temperature": 0.2,
+ "max_tokens": 4096,
+ "num_retries": 3,
+ }
+}
+```
+
+See [Custom Models](./custom-models.md) for the full `ModelSettings` reference.
+
+---
+
+## Custom observers
+
+Pass observers to receive live callbacks during a run:
+
+```python
+from exgentic import evaluate
+from exgentic.core.orchestrator.observer import Observer
+
+class PrintObserver(Observer):
+ def on_session_success(self, session, score, agent):
+ print(f"Session {session.session_id}: score={score.score}")
+
+results = evaluate(
+ benchmark="tau2",
+ agent="tool_calling",
+ num_tasks=5,
+ observers=[PrintObserver()],
+)
+```
+
+See [Observers](./observers.md) for the full interface reference.
+
+---
+
+## See also
+
+- [Custom Models](./custom-models.md) — LLM provider setup and `ModelSettings`
+- [Output Format](./output-format.md) — `RunResults` and `SessionResults` schema
+- [Observers](./observers.md) — custom event callbacks
+- [Batch Runs](./batch.md) — programmatic equivalents of batch commands
+- [CLI Reference](./cli-reference.md) — CLI alternative to the Python API
+- [docs/](./README.md) — documentation index
diff --git a/labs/AgentStream/exgentic/docs/replay-testing.md b/labs/AgentStream/exgentic/docs/replay-testing.md
new file mode 100644
index 00000000..996be3b0
--- /dev/null
+++ b/labs/AgentStream/exgentic/docs/replay-testing.md
@@ -0,0 +1,229 @@
+# Replay Testing
+
+Replay tests let you verify benchmark and agent integration end-to-end without making real API calls or running external services. A recording captures a live session; the test replays it deterministically.
+
+This is the primary testing pattern for contributors adding new benchmarks or agents.
+
+**Related docs:**
+[docs/](./README.md) · [Adding Benchmarks](./adding-benchmarks.md) · [Adding Agents](./adding-agents.md) · [Runners](./runners.md)
+
+---
+
+## How it works
+
+1. Run a live evaluation and capture the session trajectory.
+2. Store the trajectory in `tests/benchmarks/recordings//`.
+3. Write a test that replays the trajectory using `ReplayAgent` and `ReplayBenchmark`.
+4. The test verifies the session ends with the expected score — no network, no API keys, no benchmark installation required.
+
+Replay tests can be parametrized across all runner types (`direct`, `venv`, `docker`) to verify that runner isolation doesn't change session outcomes.
+
+---
+
+## Recording format
+
+Each recording lives in its own directory:
+
+```
+tests/benchmarks/recordings//
+├── recording.json # Metadata: task_id, benchmark slug, expected score
+├── trajectory.jsonl # Recorded actions and observations
+├── session.json # Session manifest
+└── results.json # Recorded session results
+```
+
+### recording.json
+
+```json
+{
+ "task_id": "retail_1",
+ "benchmark_slug": "tau2",
+ "expected_score": 1.0
+}
+```
+
+Set `expected_score` to `null` if you only want to verify that the session completes without asserting on score.
+
+### trajectory.jsonl
+
+Newline-delimited JSON. Each line is one event — either an observation (benchmark → agent) or an action (agent → benchmark):
+
+```json
+{"event": "observation", "step": 0, "initial": true, "session_id": "...", "task_id": "retail_1", "observation": {...}, "action": null}
+{"event": "action", "step": 1, "initial": false, "session_id": "...", "task_id": "retail_1", "observation": null, "action": {"name": "search", "arguments": {...}}}
+{"event": "observation", "step": 1, "initial": false, "session_id": "...", "task_id": "retail_1", "observation": {...}, "action": null}
+```
+
+---
+
+## Creating a recording
+
+Run a live evaluation and save the trajectory. The trajectory file is written automatically to the session output directory:
+
+```
+outputs//sessions//trajectory.jsonl
+outputs//sessions//results.json
+```
+
+Copy the relevant files into your recording directory:
+
+```bash
+mkdir -p tests/benchmarks/recordings/my_benchmark
+
+cp outputs//sessions//trajectory.jsonl \
+ tests/benchmarks/recordings/my_benchmark/
+
+cp outputs//sessions//results.json \
+ tests/benchmarks/recordings/my_benchmark/
+
+# Write recording.json manually
+cat > tests/benchmarks/recordings/my_benchmark/recording.json <=0.1.0
+some-other-dep==1.2.3
+```
+
+Git LFS objects are automatically skipped during install (`GIT_LFS_SKIP_SMUDGE=1`).
+
+### setup.sh
+
+Shell script for setup that can't be expressed as pip packages: cloning repositories, compiling binaries, downloading model weights, etc.
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+
+# Example: clone a dependency
+git clone --depth 1 https://github.com/example/repo /opt/repo
+```
+
+Place it in the same directory as your benchmark module. It runs after `requirements.txt` is installed.
+
+---
+
+## See also
+
+- [Adding Benchmarks](./adding-benchmarks.md) — how benchmarks declare their setup
+- [Adding Agents](./adding-agents.md) — how agents declare their setup
+- [Custom Models](./custom-models.md) — configuring the LLM behind the agent
+- [docs/](./README.md) — documentation index
diff --git a/labs/AgentStream/exgentic/examples/run_appworld.py b/labs/AgentStream/exgentic/examples/run_appworld.py
new file mode 100644
index 00000000..03600313
--- /dev/null
+++ b/labs/AgentStream/exgentic/examples/run_appworld.py
@@ -0,0 +1,24 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+## CLI usage:
+# exgentic evaluate --benchmark appworld --agent tool_calling --subset test_normal --num-tasks 3 \
+# --model gpt-4o
+## Python API usage:
+# from exgentic import RunConfig, evaluate
+# evaluate(RunConfig(benchmark="appworld", agent="tool_calling", subset="test_normal", num_tasks=3,
+# model="gpt-4o"))
+## Direct class usage (this script):
+# AppWorldBenchmark + LiteLLMToolCallingAgent
+
+from exgentic import AppWorldBenchmark, LiteLLMToolCallingAgent, evaluate
+
+
+def main() -> None:
+ benchmark = AppWorldBenchmark(subset="test_normal")
+ agent = LiteLLMToolCallingAgent(model="gpt-4o")
+ evaluate(benchmark=benchmark, agent=agent, output_dir="./outputs", num_tasks=3)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/labs/AgentStream/exgentic/examples/run_browsecomp.py b/labs/AgentStream/exgentic/examples/run_browsecomp.py
new file mode 100644
index 00000000..26176885
--- /dev/null
+++ b/labs/AgentStream/exgentic/examples/run_browsecomp.py
@@ -0,0 +1,24 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+## CLI usage:
+# exgentic evaluate --benchmark browsecompplus --agent tool_calling --subset main --num-tasks 3 \
+# --model gpt-4o
+## Python API usage:
+# from exgentic import RunConfig, evaluate
+# evaluate(RunConfig(benchmark="browsecompplus", agent="tool_calling", subset="main", num_tasks=3,
+# model="gpt-4o"))
+## Direct class usage (this script):
+# BrowseCompPlusBenchmark + LiteLLMToolCallingAgent
+
+from exgentic import BrowseCompPlusBenchmark, LiteLLMToolCallingAgent, evaluate
+
+
+def main() -> None:
+ benchmark = BrowseCompPlusBenchmark()
+ agent = LiteLLMToolCallingAgent(model="gpt-4o")
+ evaluate(benchmark=benchmark, agent=agent, output_dir="./outputs", num_tasks=3)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/labs/AgentStream/exgentic/examples/run_claude_code_on_gsm8k.py b/labs/AgentStream/exgentic/examples/run_claude_code_on_gsm8k.py
new file mode 100644
index 00000000..6e3cecb8
--- /dev/null
+++ b/labs/AgentStream/exgentic/examples/run_claude_code_on_gsm8k.py
@@ -0,0 +1,24 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+## CLI usage:
+# exgentic evaluate --benchmark gsm8k --agent claude_code --num-tasks 1 \
+# --model gpt-4o
+## Python API usage:
+# from exgentic import RunConfig, evaluate
+# evaluate(RunConfig(benchmark="gsm8k", agent="claude_code", num_tasks=1,
+# model="gpt-4o"))
+## Direct class usage (this script):
+# GSM8kBenchmark + ClaudeCodeAgent
+
+from exgentic import ClaudeCodeAgent, GSM8kBenchmark, evaluate
+
+
+def main() -> None:
+ benchmark = GSM8kBenchmark()
+ agent = ClaudeCodeAgent(model="gpt-4o")
+ evaluate(benchmark=benchmark, agent=agent, output_dir="./outputs", num_tasks=1)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/labs/AgentStream/exgentic/examples/run_claude_code_on_tau2bench.py b/labs/AgentStream/exgentic/examples/run_claude_code_on_tau2bench.py
new file mode 100644
index 00000000..b8e7b66e
--- /dev/null
+++ b/labs/AgentStream/exgentic/examples/run_claude_code_on_tau2bench.py
@@ -0,0 +1,24 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+## CLI usage:
+# exgentic evaluate --benchmark tau2 --agent claude_code --subset telecom --num-tasks 1 \
+# --model gpt-4o --set benchmark.user_simulator_model=gpt-4o
+## Python API usage:
+# from exgentic import RunConfig, evaluate
+# evaluate(RunConfig(benchmark="tau2", agent="claude_code", subset="telecom", num_tasks=1,
+# model="gpt-4o", benchmark_kwargs={"user_simulator_model": "gpt-4o"}))
+## Direct class usage (this script):
+# TAU2Benchmark + ClaudeCodeAgent
+
+from exgentic import ClaudeCodeAgent, TAU2Benchmark, evaluate
+
+
+def main() -> None:
+ benchmark = TAU2Benchmark(subset="telecom", user_simulator_model="gpt-4o")
+ agent = ClaudeCodeAgent(model="gpt-4o")
+ evaluate(benchmark=benchmark, agent=agent, output_dir="./outputs", num_tasks=1)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/labs/AgentStream/exgentic/examples/run_cli_agents.py b/labs/AgentStream/exgentic/examples/run_cli_agents.py
new file mode 100644
index 00000000..cbf518c9
--- /dev/null
+++ b/labs/AgentStream/exgentic/examples/run_cli_agents.py
@@ -0,0 +1,24 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+## CLI usage:
+# exgentic evaluate --benchmark gsm8k --agent codex_cli --num-tasks 3 \
+# --model gpt-4o
+## Python API usage:
+# from exgentic import RunConfig, evaluate
+# evaluate(RunConfig(benchmark="gsm8k", agent="codex_cli", num_tasks=3,
+# model="gpt-4o"))
+## Direct class usage (this script):
+# GSM8kBenchmark + CodexAgent
+
+from exgentic import CodexAgent, GSM8kBenchmark, evaluate
+
+
+def main() -> None:
+ benchmark = GSM8kBenchmark()
+ agent = CodexAgent(model="gpt-4o")
+ evaluate(benchmark=benchmark, agent=agent, output_dir="./outputs", num_tasks=3)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/labs/AgentStream/exgentic/examples/run_gsm8k.py b/labs/AgentStream/exgentic/examples/run_gsm8k.py
new file mode 100644
index 00000000..93466296
--- /dev/null
+++ b/labs/AgentStream/exgentic/examples/run_gsm8k.py
@@ -0,0 +1,23 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+## CLI usage:
+# exgentic evaluate --benchmark gsm8k --agent tool_calling --num-tasks 3 --model gpt-4o
+## Python API usage:
+# from exgentic import RunConfig, evaluate
+# evaluate(RunConfig(benchmark="gsm8k", agent="tool_calling", num_tasks=3,
+# model="gpt-4o"))
+## Direct class usage (this script):
+# GSM8kBenchmark + LiteLLMToolCallingAgent
+
+from exgentic import GSM8kBenchmark, LiteLLMToolCallingAgent, evaluate
+
+
+def main() -> None:
+ benchmark = GSM8kBenchmark()
+ agent = LiteLLMToolCallingAgent(model="gpt-4o")
+ evaluate(benchmark=benchmark, agent=agent, output_dir="./outputs", num_tasks=3)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/labs/AgentStream/exgentic/examples/run_hotpotqa.py b/labs/AgentStream/exgentic/examples/run_hotpotqa.py
new file mode 100644
index 00000000..eccf6563
--- /dev/null
+++ b/labs/AgentStream/exgentic/examples/run_hotpotqa.py
@@ -0,0 +1,24 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+## CLI usage:
+# exgentic evaluate --benchmark hotpotqa --agent smolagents_tool --subset distractor --num-tasks 3 \
+# --model gpt-4o --set benchmark.with_search_tools=true
+## Python API usage:
+# from exgentic import RunConfig, evaluate
+# evaluate(RunConfig(benchmark="hotpotqa", agent="smolagents_tool", subset="distractor", num_tasks=3,
+# model="gpt-4o", benchmark_kwargs={"with_search_tools": True}))
+## Direct class usage (this script):
+# HotpotQABenchmark + SmolagentToolCallingAgent
+
+from exgentic import HotpotQABenchmark, SmolagentToolCallingAgent, evaluate
+
+
+def main() -> None:
+ benchmark = HotpotQABenchmark(with_search_tools=True)
+ agent = SmolagentToolCallingAgent(model="gpt-4o")
+ evaluate(benchmark=benchmark, agent=agent, output_dir="./outputs", num_tasks=3)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/labs/AgentStream/exgentic/examples/run_openai_mcp.py b/labs/AgentStream/exgentic/examples/run_openai_mcp.py
new file mode 100644
index 00000000..08a2486b
--- /dev/null
+++ b/labs/AgentStream/exgentic/examples/run_openai_mcp.py
@@ -0,0 +1,33 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+## CLI usage:
+# exgentic evaluate --benchmark tau2 --agent openai_solo --subset retail --task 4 \
+# --model gpt-4o --set benchmark.user_simulator_model=gpt-4o
+## Python API usage:
+# from exgentic import RunConfig, evaluate
+# evaluate(RunConfig(benchmark="tau2", agent="openai_solo", subset="retail", task_ids=["4"],
+# model="gpt-4o",
+# benchmark_kwargs={"user_simulator_model": "gpt-4o"}))
+## Direct class usage (this script):
+# TAU2Benchmark + OpenAIMCPAgent
+
+from exgentic import OpenAIMCPAgent, TAU2Benchmark, evaluate
+
+
+def main() -> None:
+ benchmark = TAU2Benchmark(
+ subset="retail",
+ user_simulator_model="gpt-4o",
+ )
+ agent = OpenAIMCPAgent(model="gpt-4o")
+ evaluate(
+ benchmark=benchmark,
+ agent=agent,
+ output_dir="./outputs",
+ task_ids=["4"],
+ )
+
+
+if __name__ == "__main__":
+ main()
diff --git a/labs/AgentStream/exgentic/examples/run_smol.py b/labs/AgentStream/exgentic/examples/run_smol.py
new file mode 100644
index 00000000..2b616b68
--- /dev/null
+++ b/labs/AgentStream/exgentic/examples/run_smol.py
@@ -0,0 +1,24 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+## CLI usage:
+# exgentic evaluate --benchmark tau2 --agent smolagents_code --subset retail --num-tasks 30 \
+# --model gpt-4o --set benchmark.user_simulator_model=gpt-4o
+## Python API usage:
+# from exgentic import RunConfig, evaluate
+# evaluate(RunConfig(benchmark="tau2", agent="smolagents_code", subset="retail", num_tasks=30,
+# model="gpt-4o", benchmark_kwargs={"user_simulator_model": "gpt-4o"}))
+## Direct class usage (this script):
+# TAU2Benchmark + SmolagentCodeAgent
+
+from exgentic import SmolagentCodeAgent, TAU2Benchmark, evaluate
+
+
+def main() -> None:
+ benchmark = TAU2Benchmark(subset="retail", user_simulator_model="gpt-4o")
+ agent = SmolagentCodeAgent(model="gpt-4o")
+ evaluate(benchmark=benchmark, agent=agent, output_dir="./outputs", num_tasks=30)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/labs/AgentStream/exgentic/examples/run_swebench.py b/labs/AgentStream/exgentic/examples/run_swebench.py
new file mode 100644
index 00000000..0d7d9492
--- /dev/null
+++ b/labs/AgentStream/exgentic/examples/run_swebench.py
@@ -0,0 +1,24 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+## CLI usage:
+# exgentic evaluate --benchmark swebench --agent tool_calling --num-tasks 30 \
+# --model gpt-4o
+## Python API usage:
+# from exgentic import RunConfig, evaluate
+# evaluate(RunConfig(benchmark="swebench", agent="tool_calling", num_tasks=30,
+# model="gpt-4o"))
+## Direct class usage (this script):
+# SWEBenchBenchmark + LiteLLMToolCallingAgent
+
+from exgentic import LiteLLMToolCallingAgent, SWEBenchBenchmark, evaluate
+
+
+def main() -> None:
+ benchmark = SWEBenchBenchmark()
+ agent = LiteLLMToolCallingAgent(model="gpt-4o")
+ evaluate(benchmark=benchmark, agent=agent, output_dir="./outputs", num_tasks=30)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/labs/AgentStream/exgentic/examples/run_taubench.py b/labs/AgentStream/exgentic/examples/run_taubench.py
new file mode 100644
index 00000000..c2a2f1bd
--- /dev/null
+++ b/labs/AgentStream/exgentic/examples/run_taubench.py
@@ -0,0 +1,24 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+## CLI usage:
+# exgentic evaluate --benchmark tau2 --agent tool_calling --subset retail --num-tasks 1 \
+# --model gpt-4o --set benchmark.user_simulator_model=gpt-4o
+## Python API usage:
+# from exgentic import RunConfig, evaluate
+# evaluate(RunConfig(benchmark="tau2", agent="tool_calling", subset="retail", num_tasks=1,
+# model="gpt-4o", benchmark_kwargs={"user_simulator_model": "gpt-4o"}))
+## Direct class usage (this script):
+# TAU2Benchmark + LiteLLMToolCallingAgent
+
+from exgentic import LiteLLMToolCallingAgent, TAU2Benchmark, evaluate
+
+
+def main() -> None:
+ benchmark = TAU2Benchmark(subset="retail", user_simulator_model="gpt-4o")
+ agent = LiteLLMToolCallingAgent(model="gpt-4o")
+ evaluate(benchmark=benchmark, agent=agent, output_dir="./outputs", num_tasks=1)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/labs/AgentStream/exgentic/examples/simple_test_agent/adapter.py b/labs/AgentStream/exgentic/examples/simple_test_agent/adapter.py
new file mode 100644
index 00000000..1501ea4a
--- /dev/null
+++ b/labs/AgentStream/exgentic/examples/simple_test_agent/adapter.py
@@ -0,0 +1,54 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from typing import Any, Dict, List, Optional
+
+from exgentic.core.agent import Agent
+from exgentic.core.agent_instance import AgentInstance
+from exgentic.core.types import ModelSettings
+
+
+class SimpleTestAgentInstance(AgentInstance):
+ """Simple test agent that responds with basic actions."""
+
+ def __init__(self, session_id: str, task: str, context: Dict[str, Any], actions: List[str]):
+ super().__init__(session_id)
+ self.task = task
+ self.context = context or {}
+ self.actions = actions
+ self.step_count = 0
+
+ def react(self, observation: Optional[str]) -> Optional[str]:
+ """React to observation with simple response."""
+ self.step_count += 1
+
+ if observation is None:
+ # First step
+ return f"Starting task: {self.task}"
+
+ # Simple logic: respond a few times then finish
+ if self.step_count <= 2:
+ return f"Responding to: {observation}"
+ # Signal completion
+ return None
+
+ def close(self):
+ pass
+
+
+class SimpleTestAgent(Agent):
+ """Agent factory that creates simple test agents."""
+
+ display_name: str = "Simple Test Agent"
+ slug_name: str = "simple_test"
+
+ def __init__(self, model_settings: ModelSettings | None = None) -> None:
+ if model_settings is not None and not isinstance(model_settings, ModelSettings):
+ raise ValueError("model_settings must be a ModelSettings instance.")
+ self.model_settings = model_settings
+
+ def assign(self, task: str, context: Dict[str, Any], actions: List[str], session_id: str) -> AgentInstance:
+ return SimpleTestAgentInstance(session_id, task, context, actions)
+
+ def get_models_names(self) -> List[str]: # type: ignore[override]
+ return []
diff --git a/labs/AgentStream/exgentic/examples/simple_test_agent/setup.sh b/labs/AgentStream/exgentic/examples/simple_test_agent/setup.sh
new file mode 100644
index 00000000..5ddf8ed1
--- /dev/null
+++ b/labs/AgentStream/exgentic/examples/simple_test_agent/setup.sh
@@ -0,0 +1,3 @@
+#!/bin/bash
+# No additional dependencies needed for simple test agent
+echo "Simple test agent - no setup required"
diff --git a/labs/AgentStream/exgentic/misc/assets/cli.png b/labs/AgentStream/exgentic/misc/assets/cli.png
new file mode 100644
index 00000000..fb4f5ad1
Binary files /dev/null and b/labs/AgentStream/exgentic/misc/assets/cli.png differ
diff --git a/labs/AgentStream/exgentic/misc/assets/exgentic_banner_black.png b/labs/AgentStream/exgentic/misc/assets/exgentic_banner_black.png
new file mode 100644
index 00000000..30030776
Binary files /dev/null and b/labs/AgentStream/exgentic/misc/assets/exgentic_banner_black.png differ
diff --git a/labs/AgentStream/exgentic/misc/assets/exgentic_banner_black_no_background.png b/labs/AgentStream/exgentic/misc/assets/exgentic_banner_black_no_background.png
new file mode 100644
index 00000000..e5286cc7
Binary files /dev/null and b/labs/AgentStream/exgentic/misc/assets/exgentic_banner_black_no_background.png differ
diff --git a/labs/AgentStream/exgentic/misc/assets/exgentic_banner_white.png b/labs/AgentStream/exgentic/misc/assets/exgentic_banner_white.png
new file mode 100644
index 00000000..f9539385
Binary files /dev/null and b/labs/AgentStream/exgentic/misc/assets/exgentic_banner_white.png differ
diff --git a/labs/AgentStream/exgentic/misc/assets/exgentic_banner_white_no_background.png b/labs/AgentStream/exgentic/misc/assets/exgentic_banner_white_no_background.png
new file mode 100644
index 00000000..8f272db3
Binary files /dev/null and b/labs/AgentStream/exgentic/misc/assets/exgentic_banner_white_no_background.png differ
diff --git a/labs/AgentStream/exgentic/misc/assets/exgentic_light.png b/labs/AgentStream/exgentic/misc/assets/exgentic_light.png
new file mode 100644
index 00000000..844b2302
Binary files /dev/null and b/labs/AgentStream/exgentic/misc/assets/exgentic_light.png differ
diff --git a/labs/AgentStream/exgentic/misc/assets/gui.png b/labs/AgentStream/exgentic/misc/assets/gui.png
new file mode 100644
index 00000000..5997c98a
Binary files /dev/null and b/labs/AgentStream/exgentic/misc/assets/gui.png differ
diff --git a/labs/AgentStream/exgentic/misc/assets/icon.png b/labs/AgentStream/exgentic/misc/assets/icon.png
new file mode 100644
index 00000000..8e838737
Binary files /dev/null and b/labs/AgentStream/exgentic/misc/assets/icon.png differ
diff --git a/labs/AgentStream/exgentic/misc/assets/icon_black.png b/labs/AgentStream/exgentic/misc/assets/icon_black.png
new file mode 100644
index 00000000..72092af3
Binary files /dev/null and b/labs/AgentStream/exgentic/misc/assets/icon_black.png differ
diff --git a/labs/AgentStream/exgentic/misc/security/requirements/README.md b/labs/AgentStream/exgentic/misc/security/requirements/README.md
new file mode 100644
index 00000000..05949575
--- /dev/null
+++ b/labs/AgentStream/exgentic/misc/security/requirements/README.md
@@ -0,0 +1,39 @@
+# Requirements Files
+
+This directory contains frozen requirements for each virtual environment.
+
+## Directories
+
+Each environment has its own directory with a `requirements.txt` file:
+
+- `core/requirements.txt` - Core environment with all optional dependencies
+- `gsm8k/requirements.txt` - GSM8K benchmark environment
+- `hotpotqa/requirements.txt` - HotpotQA benchmark environment
+- `appworld/requirements.txt` - AppWorld benchmark environment
+- `browsecompplus/requirements.txt` - BrowseComp+ benchmark environment
+- `swebench/requirements.txt` - SWE-bench benchmark environment
+- `tau2/requirements.txt` - TAU-2 benchmark environment
+
+## Usage
+
+To recreate an environment from a requirements file:
+
+```bash
+# Create a new virtual environment
+uv venv .venv --python 3.11
+
+# Activate it
+source .venv/bin/activate
+
+# Install from requirements
+uv pip install -r misc/security/requirements/core/requirements.txt
+```
+
+## Notes
+
+- These files are generated automatically by `misc/security/setup_environments.sh`
+- They represent the exact package versions installed in each environment
+- Only PyPI packages are included (no git or local installs)
+- Regenerate by re-running the setup script
+
+## Generation Date
diff --git a/labs/AgentStream/exgentic/misc/security/requirements/appworld/requirements.txt b/labs/AgentStream/exgentic/misc/security/requirements/appworld/requirements.txt
new file mode 100644
index 00000000..5a8934bc
--- /dev/null
+++ b/labs/AgentStream/exgentic/misc/security/requirements/appworld/requirements.txt
@@ -0,0 +1,195 @@
+aiofiles==25.1.0
+aiohappyeyeballs==2.6.1
+aiohttp==3.13.3
+aiosignal==1.4.0
+annotated-doc==0.0.4
+annotated-types==0.7.0
+anyio==4.12.1
+apscheduler==3.11.2
+asttokens==3.0.1
+attrs==25.4.0
+azure-core==1.38.2
+azure-identity==1.25.2
+azure-storage-blob==12.28.0
+backoff==2.2.1
+bidict==0.23.1
+boto3==1.40.76
+botocore==1.40.76
+certifi==2026.2.25
+cffi==2.0.0
+cfgv==3.5.0
+charset-normalizer==3.4.4
+click==8.3.1
+cloudpickle==3.1.2
+codespell==2.4.1
+colorama==0.4.6
+contourpy==1.3.3
+croniter==6.0.0
+cryptography==46.0.5
+cycler==0.12.1
+decorator==5.2.1
+detect-secrets==1.5.0
+diskcache==5.6.3
+distlib==0.4.0
+distro==1.9.0
+dnspython==2.8.0
+docutils==0.22.4
+email-validator==2.3.0
+execnet==2.1.2
+executing==2.2.1
+faker==40.5.1
+fastapi==0.135.1
+fastapi-login==1.10.3
+fastapi-sso==0.16.0
+fastuuid==0.14.0
+filelock==3.25.0
+fonttools==4.61.1
+freezegun==1.5.1
+frozenlist==1.8.0
+fsspec==2026.2.0
+googleapis-common-protos==1.72.0
+griffe==1.15.0
+grpcio==1.78.0
+gunicorn==23.0.0
+h11==0.16.0
+hf-xet==1.3.2
+httpcore==1.0.9
+httptools==0.7.1
+httpx==0.28.1
+httpx-sse==0.4.3
+huggingface-hub==0.36.2
+identify==2.6.17
+idna==3.11
+ifaddr==0.2.0
+importlib-metadata==8.7.1
+inflection==0.5.1
+iniconfig==2.3.0
+ipython==9.10.0
+ipython-pygments-lexers==1.1.1
+isodate==0.7.2
+itsdangerous==2.2.0
+jedi==0.19.2
+jinja2==3.1.6
+jiter==0.13.0
+jmespath==1.1.0
+json-schema-to-pydantic==0.4.9
+jsonref==1.1.0
+jsonschema==4.26.0
+jsonschema-specifications==2025.9.1
+kiwisolver==1.4.9
+libcst==1.8.6
+litellm==1.82.0
+litellm-enterprise==0.1.33
+litellm-proxy-extras==0.4.50
+markdown-it-py==4.0.0
+markdown2==2.5.5
+markupsafe==3.0.3
+matplotlib==3.10.8
+matplotlib-inline==0.2.1
+mcp==1.26.0
+mdurl==0.1.2
+msal==1.35.0
+msal-extensions==1.3.1
+multidict==6.7.1
+munch==4.0.0
+nicegui==3.8.0
+nodeenv==1.10.0
+numpy==2.4.2
+oauthlib==3.3.1
+openai==2.24.0
+openai-agents==0.10.3
+opentelemetry-api==1.39.1
+opentelemetry-exporter-otlp-proto-common==1.39.1
+opentelemetry-exporter-otlp-proto-grpc==1.39.1
+opentelemetry-exporter-otlp-proto-http==1.39.1
+opentelemetry-proto==1.39.1
+opentelemetry-sdk==1.39.1
+opentelemetry-semantic-conventions==0.60b1
+opentelemetry-semantic-conventions-ai==0.4.15
+orjson==3.11.7
+packaging==26.0
+pandas==3.0.1
+parso==0.8.6
+patsy==1.0.2
+pendulum==3.2.0
+pexpect==4.9.0
+pillow==12.1.1
+platformdirs==4.9.2
+pluggy==1.6.0
+polars==1.38.1
+polars-runtime-32==1.38.1
+polyfactory==3.3.0
+pre-commit==4.5.1
+prompt-toolkit==3.0.52
+propcache==0.4.1
+protobuf==6.33.5
+psutil==7.2.2
+ptyprocess==0.7.0
+pure-eval==0.2.3
+pycparser==3.0
+pydantic==2.12.5
+pydantic-core==2.41.5
+pydantic-extra-types==2.11.0
+pydantic-settings==2.13.1
+pygments==2.19.2
+pyjwt==2.11.0
+pynacl==1.6.2
+pyparsing==3.3.2
+pyroscope-io==0.8.16
+pytest==9.0.2
+pytest-asyncio==1.3.0
+pytest-mock==3.15.1
+pytest-xdist==3.8.0
+python-dateutil==2.9.0.post0
+python-discovery==1.1.0
+python-dotenv==1.2.2
+python-engineio==4.13.1
+python-multipart==0.0.22
+python-socketio==5.16.1
+pytz==2026.1.post1
+pyyaml==6.0.3
+redis==7.2.1
+referencing==0.37.0
+regex==2026.2.28
+requests==2.32.5
+rich==14.3.3
+rich-click==1.9.7
+rpds-py==0.30.0
+rq==2.7.0
+ruff==0.15.4
+s3transfer==0.14.0
+scipy==1.17.1
+shellingham==1.5.4
+simple-websocket==1.1.0
+six==1.17.0
+smolagents==1.24.0
+sniffio==1.3.1
+soundfile==0.12.1
+sqlalchemy==2.0.48
+sqlalchemy-utils==0.42.1
+sqlmodel==0.0.37
+sse-starlette==3.3.2
+stack-data==0.6.3
+starlette==0.52.1
+statsmodels==0.14.6
+tiktoken==0.12.0
+tokenizers==0.22.2
+tqdm==4.67.3
+traitlets==5.14.3
+typer==0.24.1
+types-requests==2.32.4.20260107
+typing-extensions==4.15.0
+typing-inspection==0.4.2
+tzdata==2025.3
+tzlocal==5.3.1
+urllib3==2.6.3
+uvicorn==0.41.0
+uvloop==0.21.0
+virtualenv==21.1.0
+watchfiles==1.1.1
+wcwidth==0.6.0
+websockets==15.0.1
+wsproto==1.3.2
+xxhash==3.6.0
+yarl==1.23.0
+zipp==3.23.0
diff --git a/labs/AgentStream/exgentic/misc/security/requirements/browsecompplus/requirements.txt b/labs/AgentStream/exgentic/misc/security/requirements/browsecompplus/requirements.txt
new file mode 100644
index 00000000..c4cc59be
--- /dev/null
+++ b/labs/AgentStream/exgentic/misc/security/requirements/browsecompplus/requirements.txt
@@ -0,0 +1,261 @@
+accelerate==1.12.0
+aiofile==3.9.0
+aiofiles==23.2.1
+aiohappyeyeballs==2.6.1
+aiohttp==3.13.3
+aiosignal==1.4.0
+annotated-doc==0.0.4
+annotated-types==0.7.0
+anthropic==0.84.0
+anyio==4.12.1
+apscheduler==3.11.2
+attrs==25.4.0
+audioread==3.1.0
+authlib==1.6.9
+av==16.1.0
+azure-core==1.38.2
+azure-identity==1.25.2
+azure-storage-blob==12.28.0
+backoff==2.2.1
+backports-tarfile==1.2.0
+beartype==0.22.9
+beautifulsoup4==4.14.3
+bidict==0.23.1
+blinker==1.9.0
+boto3==1.40.76
+botocore==1.40.76
+cachetools==7.0.2
+caio==0.9.25
+certifi==2026.2.25
+cffi==2.0.0
+cfgv==3.5.0
+charset-normalizer==3.4.4
+click==8.3.1
+cloudpickle==3.1.2
+codespell==2.4.1
+colorama==0.4.6
+contourpy==1.3.3
+croniter==6.0.0
+cryptography==46.0.5
+cycler==0.12.1
+cyclopts==4.6.0
+cython==3.2.4
+dashscope==1.25.13
+datasets==4.6.1
+decorator==5.2.1
+detect-secrets==1.5.0
+dill==0.4.0
+diskcache==5.6.3
+distlib==0.4.0
+distro==1.9.0
+dnspython==2.8.0
+docstring-parser==0.17.0
+docutils==0.22.4
+email-validator==2.3.0
+eval-type-backport==0.3.1
+exceptiongroup==1.3.1
+faiss-cpu==1.13.2
+fastapi==0.135.1
+fastapi-sso==0.21.0
+fastmcp==3.1.0
+fastuuid==0.14.0
+ffmpy==1.0.0
+filelock==3.25.0
+flask==3.1.3
+flatbuffers==25.12.19
+fonttools==4.61.1
+frozenlist==1.8.0
+fsspec==2026.2.0
+google-auth==2.48.0
+google-genai==1.65.0
+googleapis-common-protos==1.72.0
+gradio-client==1.8.0
+griffe==1.15.0
+groovy==0.1.2
+grpcio==1.78.0
+gunicorn==23.0.0
+h11==0.16.0
+hf-transfer==0.1.9
+hf-xet==1.3.2
+httpcore==1.0.9
+httptools==0.7.1
+httpx==0.28.1
+httpx-sse==0.4.3
+huggingface-hub==0.36.2
+identify==2.6.17
+idna==3.11
+ifaddr==0.2.0
+importlib-metadata==8.7.1
+iniconfig==2.3.0
+isodate==0.7.2
+itsdangerous==2.2.0
+jaraco-classes==3.4.0
+jaraco-context==6.1.0
+jaraco-functools==4.4.0
+jieba==0.42.1
+jinja2==3.1.6
+jiter==0.13.0
+jmespath==1.1.0
+joblib==1.5.3
+json-schema-to-pydantic==0.4.9
+json5==0.13.0
+jsonlines==4.0.0
+jsonref==1.1.0
+jsonschema==4.26.0
+jsonschema-path==0.4.4
+jsonschema-specifications==2025.9.1
+keyring==25.7.0
+kiwisolver==1.4.9
+lazy-loader==0.4
+librosa==0.11.0
+litellm==1.82.0
+litellm-enterprise==0.1.33
+litellm-proxy-extras==0.4.50
+llvmlite==0.46.0
+lxml==6.0.2
+markdown-it-py==4.0.0
+markdown2==2.5.5
+markupsafe==3.0.3
+matplotlib==3.10.8
+mcp==1.26.0
+mdurl==0.1.2
+modelscope-studio==1.1.7
+more-itertools==10.8.0
+mpmath==1.3.0
+msal==1.35.0
+msal-extensions==1.3.1
+msgpack==1.1.2
+multidict==6.7.1
+multiprocess==0.70.18
+networkx==3.6.1
+nicegui==3.8.0
+nodeenv==1.10.0
+numba==0.64.0
+numpy==2.4.2
+oauthlib==3.3.1
+onnxruntime==1.24.2
+openai==2.24.0
+openai-agents==0.10.3
+openapi-pydantic==0.5.1
+opentelemetry-api==1.39.1
+opentelemetry-exporter-otlp-proto-common==1.39.1
+opentelemetry-exporter-otlp-proto-grpc==1.39.1
+opentelemetry-exporter-otlp-proto-http==1.39.1
+opentelemetry-proto==1.39.1
+opentelemetry-sdk==1.39.1
+opentelemetry-semantic-conventions==0.60b1
+opentelemetry-semantic-conventions-ai==0.4.15
+orjson==3.11.7
+packaging==26.0
+pandas==2.3.3
+pathable==0.5.0
+patsy==1.0.2
+pdfminer-six==20251230
+pdfplumber==0.11.9
+peft==0.18.1
+pillow==12.1.1
+platformdirs==4.9.2
+pluggy==1.6.0
+polars==1.38.1
+polars-runtime-32==1.38.1
+pooch==1.9.0
+pre-commit==4.5.1
+propcache==0.4.1
+protobuf==6.33.5
+psutil==7.2.2
+py-key-value-aio==0.4.4
+pyarrow==23.0.1
+pyasn1==0.6.2
+pyasn1-modules==0.4.2
+pycparser==3.0
+pydantic==2.12.5
+pydantic-core==2.41.5
+pydantic-settings==2.13.1
+pydub==0.25.1
+pygments==2.19.2
+pyjnius==1.7.0
+pyjwt==2.11.0
+pynacl==1.6.2
+pyngrok==7.5.0
+pyparsing==3.3.2
+pypdfium2==5.5.0
+pyperclip==1.11.0
+pyroscope-io==0.8.16
+pyserini==1.2.0
+pytest==9.0.2
+pytest-asyncio==1.3.0
+pytest-mock==3.15.1
+python-dateutil==2.9.0.post0
+python-discovery==1.1.0
+python-docx==1.2.0
+python-dotenv==1.2.2
+python-engineio==4.13.1
+python-multipart==0.0.22
+python-pptx==1.0.2
+python-socketio==5.16.1
+pytz==2026.1.post1
+pyyaml==6.0.3
+qwen-agent==0.0.27
+qwen-omni-utils==0.0.8
+rank-bm25==0.2.2
+redis==7.2.1
+referencing==0.37.0
+regex==2026.2.28
+requests==2.32.5
+rich==14.3.3
+rich-click==1.9.7
+rich-rst==1.3.2
+rpds-py==0.30.0
+rq==2.7.0
+rsa==4.9.1
+ruff==0.15.4
+s3transfer==0.14.0
+safehttpx==0.1.7
+safetensors==0.7.0
+scikit-learn==1.8.0
+scipy==1.17.1
+semantic-version==2.10.0
+sentencepiece==0.2.1
+shellingham==1.5.4
+simple-websocket==1.1.0
+six==1.17.0
+smolagents==1.24.0
+sniffio==1.3.1
+snowballstemmer==3.0.1
+soundfile==0.12.1
+soupsieve==2.8.3
+soxr==1.0.0
+sse-starlette==3.3.2
+starlette==0.52.1
+statsmodels==0.14.6
+sympy==1.14.0
+tabulate==0.9.0
+tenacity==9.1.4
+threadpoolctl==3.6.0
+tiktoken==0.12.0
+tokenizers==0.22.2
+tomlkit==0.13.3
+torch==2.10.0
+torchvision==0.25.0
+tqdm==4.67.3
+transformers==4.57.6
+typer==0.24.1
+types-requests==2.32.4.20260107
+typing-extensions==4.15.0
+typing-inspection==0.4.2
+tzdata==2025.3
+tzlocal==5.3.1
+uncalled-for==0.2.0
+urllib3==2.6.3
+uvicorn==0.41.0
+uvloop==0.21.0
+virtualenv==21.1.0
+watchfiles==1.1.1
+websocket-client==1.9.0
+websockets==16.0
+werkzeug==3.1.6
+wsproto==1.3.2
+xlsxwriter==3.2.9
+xxhash==3.6.0
+yarl==1.23.0
+zipp==3.23.0
diff --git a/labs/AgentStream/exgentic/misc/security/requirements/core/requirements.txt b/labs/AgentStream/exgentic/misc/security/requirements/core/requirements.txt
new file mode 100644
index 00000000..1917cef5
--- /dev/null
+++ b/labs/AgentStream/exgentic/misc/security/requirements/core/requirements.txt
@@ -0,0 +1,160 @@
+aiofiles==25.1.0
+aiohappyeyeballs==2.6.1
+aiohttp==3.13.3
+aiosignal==1.4.0
+annotated-doc==0.0.4
+annotated-types==0.7.0
+anyio==4.12.1
+apscheduler==3.11.2
+attrs==25.4.0
+azure-core==1.38.2
+azure-identity==1.25.2
+azure-storage-blob==12.28.0
+backoff==2.2.1
+bidict==0.23.1
+boto3==1.40.76
+botocore==1.40.76
+certifi==2026.2.25
+cffi==2.0.0
+cfgv==3.5.0
+charset-normalizer==3.4.4
+click==8.3.1
+cloudpickle==3.1.2
+codespell==2.4.1
+colorama==0.4.6
+contourpy==1.3.3
+croniter==6.0.0
+cryptography==46.0.5
+cycler==0.12.1
+detect-secrets==1.5.0
+diskcache==5.6.3
+distlib==0.4.0
+distro==1.9.0
+dnspython==2.8.0
+docutils==0.22.4
+email-validator==2.3.0
+fastapi==0.135.1
+fastapi-sso==0.16.0
+fastuuid==0.14.0
+filelock==3.25.0
+fonttools==4.61.1
+frozenlist==1.8.0
+fsspec==2026.2.0
+googleapis-common-protos==1.72.0
+griffe==1.15.0
+grpcio==1.78.0
+gunicorn==23.0.0
+h11==0.16.0
+hf-xet==1.3.2
+httpcore==1.0.9
+httptools==0.7.1
+httpx==0.28.1
+httpx-sse==0.4.3
+huggingface-hub==0.36.2
+identify==2.6.17
+idna==3.11
+ifaddr==0.2.0
+importlib-metadata==8.7.1
+iniconfig==2.3.0
+isodate==0.7.2
+itsdangerous==2.2.0
+jinja2==3.1.6
+jiter==0.13.0
+jmespath==1.1.0
+json-schema-to-pydantic==0.4.9
+jsonschema==4.26.0
+jsonschema-specifications==2025.9.1
+kiwisolver==1.4.9
+litellm==1.82.0
+litellm-enterprise==0.1.33
+litellm-proxy-extras==0.4.50
+markdown-it-py==4.0.0
+markdown2==2.5.5
+markupsafe==3.0.3
+matplotlib==3.10.8
+mcp==1.26.0
+mdurl==0.1.2
+msal==1.35.0
+msal-extensions==1.3.1
+multidict==6.7.1
+nicegui==3.8.0
+nodeenv==1.10.0
+numpy==2.4.2
+oauthlib==3.3.1
+openai==2.24.0
+openai-agents==0.10.3
+opentelemetry-api==1.39.1
+opentelemetry-exporter-otlp-proto-common==1.39.1
+opentelemetry-exporter-otlp-proto-grpc==1.39.1
+opentelemetry-exporter-otlp-proto-http==1.39.1
+opentelemetry-proto==1.39.1
+opentelemetry-sdk==1.39.1
+opentelemetry-semantic-conventions==0.60b1
+opentelemetry-semantic-conventions-ai==0.4.15
+orjson==3.11.7
+packaging==26.0
+pandas==3.0.1
+patsy==1.0.2
+pillow==12.1.1
+platformdirs==4.9.2
+pluggy==1.6.0
+polars==1.38.1
+polars-runtime-32==1.38.1
+pre-commit==4.5.1
+propcache==0.4.1
+protobuf==6.33.5
+pycparser==3.0
+pydantic==2.12.5
+pydantic-core==2.41.5
+pydantic-settings==2.13.1
+pygments==2.19.2
+pyjwt==2.11.0
+pynacl==1.6.2
+pyparsing==3.3.2
+pyroscope-io==0.8.16
+pytest==9.0.2
+pytest-asyncio==1.3.0
+pytest-mock==3.15.1
+python-dateutil==2.9.0.post0
+python-discovery==1.1.0
+python-dotenv==1.2.2
+python-engineio==4.13.1
+python-multipart==0.0.22
+python-socketio==5.16.1
+pytz==2026.1.post1
+pyyaml==6.0.3
+redis==7.2.1
+referencing==0.37.0
+regex==2026.2.28
+requests==2.32.5
+rich==14.3.3
+rich-click==1.9.7
+rpds-py==0.30.0
+rq==2.7.0
+ruff==0.15.4
+s3transfer==0.14.0
+scipy==1.17.1
+simple-websocket==1.1.0
+six==1.17.0
+smolagents==1.24.0
+sniffio==1.3.1
+soundfile==0.12.1
+sse-starlette==3.3.2
+starlette==0.52.1
+statsmodels==0.14.6
+tiktoken==0.12.0
+tokenizers==0.22.2
+tqdm==4.67.3
+types-requests==2.32.4.20260107
+typing-extensions==4.15.0
+typing-inspection==0.4.2
+tzlocal==5.3.1
+urllib3==2.6.3
+uvicorn==0.41.0
+uvloop==0.21.0
+virtualenv==21.1.0
+watchfiles==1.1.1
+websockets==15.0.1
+wsproto==1.3.2
+yarl==1.23.0
+zipp==3.23.0
diff --git a/labs/AgentStream/exgentic/misc/security/requirements/gsm8k/requirements.txt b/labs/AgentStream/exgentic/misc/security/requirements/gsm8k/requirements.txt
new file mode 100644
index 00000000..13c240c8
--- /dev/null
+++ b/labs/AgentStream/exgentic/misc/security/requirements/gsm8k/requirements.txt
@@ -0,0 +1,165 @@
+aiofiles==25.1.0
+aiohappyeyeballs==2.6.1
+aiohttp==3.13.3
+aiosignal==1.4.0
+annotated-doc==0.0.4
+annotated-types==0.7.0
+anyio==4.12.1
+apscheduler==3.11.2
+attrs==25.4.0
+azure-core==1.38.2
+azure-identity==1.25.2
+azure-storage-blob==12.28.0
+backoff==2.2.1
+bidict==0.23.1
+boto3==1.40.76
+botocore==1.40.76
+certifi==2026.2.25
+cffi==2.0.0
+cfgv==3.5.0
+charset-normalizer==3.4.4
+click==8.3.1
+cloudpickle==3.1.2
+codespell==2.4.1
+colorama==0.4.6
+contourpy==1.3.3
+croniter==6.0.0
+cryptography==46.0.5
+cycler==0.12.1
+datasets==4.6.1
+detect-secrets==1.5.0
+dill==0.4.0
+diskcache==5.6.3
+distlib==0.4.0
+distro==1.9.0
+dnspython==2.8.0
+docutils==0.22.4
+email-validator==2.3.0
+fastapi==0.135.1
+fastapi-sso==0.16.0
+fastuuid==0.14.0
+filelock==3.25.0
+fonttools==4.61.1
+frozenlist==1.8.0
+fsspec==2026.2.0
+googleapis-common-protos==1.72.0
+griffe==1.15.0
+grpcio==1.78.0
+gunicorn==23.0.0
+h11==0.16.0
+hf-xet==1.3.2
+httpcore==1.0.9
+httptools==0.7.1
+httpx==0.28.1
+httpx-sse==0.4.3
+huggingface-hub==0.36.2
+identify==2.6.17
+idna==3.11
+ifaddr==0.2.0
+importlib-metadata==8.7.1
+iniconfig==2.3.0
+isodate==0.7.2
+itsdangerous==2.2.0
+jinja2==3.1.6
+jiter==0.13.0
+jmespath==1.1.0
+json-schema-to-pydantic==0.4.9
+jsonschema==4.26.0
+jsonschema-specifications==2025.9.1
+kiwisolver==1.4.9
+litellm==1.82.0
+litellm-enterprise==0.1.33
+litellm-proxy-extras==0.4.50
+markdown-it-py==4.0.0
+markdown2==2.5.5
+markupsafe==3.0.3
+matplotlib==3.10.8
+mcp==1.26.0
+mdurl==0.1.2
+msal==1.35.0
+msal-extensions==1.3.1
+multidict==6.7.1
+multiprocess==0.70.18
+nicegui==3.8.0
+nodeenv==1.10.0
+numpy==2.4.2
+oauthlib==3.3.1
+openai==2.24.0
+openai-agents==0.10.3
+opentelemetry-api==1.39.1
+opentelemetry-exporter-otlp-proto-common==1.39.1
+opentelemetry-exporter-otlp-proto-grpc==1.39.1
+opentelemetry-exporter-otlp-proto-http==1.39.1
+opentelemetry-proto==1.39.1
+opentelemetry-sdk==1.39.1
+opentelemetry-semantic-conventions==0.60b1
+opentelemetry-semantic-conventions-ai==0.4.15
+orjson==3.11.7
+packaging==26.0
+pandas==3.0.1
+patsy==1.0.2
+pillow==12.1.1
+platformdirs==4.9.2
+pluggy==1.6.0
+polars==1.38.1
+polars-runtime-32==1.38.1
+pre-commit==4.5.1
+propcache==0.4.1
+protobuf==6.33.5
+pyarrow==23.0.1
+pycparser==3.0
+pydantic==2.12.5
+pydantic-core==2.41.5
+pydantic-settings==2.13.1
+pygments==2.19.2
+pyjwt==2.11.0
+pynacl==1.6.2
+pyparsing==3.3.2
+pyroscope-io==0.8.16
+pytest==9.0.2
+pytest-asyncio==1.3.0
+pytest-mock==3.15.1
+python-dateutil==2.9.0.post0
+python-discovery==1.1.0
+python-dotenv==1.2.2
+python-engineio==4.13.1
+python-multipart==0.0.22
+python-socketio==5.16.1
+pytz==2026.1.post1
+pyyaml==6.0.3
+redis==7.2.1
+referencing==0.37.0
+regex==2026.2.28
+requests==2.32.5
+rich==14.3.3
+rich-click==1.9.7
+rpds-py==0.30.0
+rq==2.7.0
+ruff==0.15.4
+s3transfer==0.14.0
+scipy==1.17.1
+simple-websocket==1.1.0
+six==1.17.0
+smolagents==1.24.0
+sniffio==1.3.1
+soundfile==0.12.1
+sse-starlette==3.3.2
+starlette==0.52.1
+statsmodels==0.14.6
+tiktoken==0.12.0
+tokenizers==0.22.2
+tqdm==4.67.3
+types-requests==2.32.4.20260107
+typing-extensions==4.15.0
+typing-inspection==0.4.2
+tzlocal==5.3.1
+urllib3==2.6.3
+uvicorn==0.41.0
+uvloop==0.21.0
+virtualenv==21.1.0
+watchfiles==1.1.1
+websockets==15.0.1
+wsproto==1.3.2
+xxhash==3.6.0
+yarl==1.23.0
+zipp==3.23.0
diff --git a/labs/AgentStream/exgentic/misc/security/requirements/hotpotqa/requirements.txt b/labs/AgentStream/exgentic/misc/security/requirements/hotpotqa/requirements.txt
new file mode 100644
index 00000000..8a53efe6
--- /dev/null
+++ b/labs/AgentStream/exgentic/misc/security/requirements/hotpotqa/requirements.txt
@@ -0,0 +1,190 @@
+aiofile==3.9.0
+aiofiles==25.1.0
+aiohappyeyeballs==2.6.1
+aiohttp==3.13.3
+aiosignal==1.4.0
+annotated-doc==0.0.4
+annotated-types==0.7.0
+anyio==4.12.1
+apscheduler==3.11.2
+attrs==25.4.0
+authlib==1.6.9
+azure-core==1.38.2
+azure-identity==1.25.2
+azure-storage-blob==12.28.0
+backoff==2.2.1
+backports-tarfile==1.2.0
+beartype==0.22.9
+bidict==0.23.1
+boto3==1.40.76
+botocore==1.40.76
+cachetools==7.0.2
+caio==0.9.25
+certifi==2026.2.25
+cffi==2.0.0
+cfgv==3.5.0
+charset-normalizer==3.4.4
+click==8.3.1
+cloudpickle==3.1.2
+codespell==2.4.1
+colorama==0.4.6
+contourpy==1.3.3
+croniter==6.0.0
+cryptography==46.0.5
+cycler==0.12.1
+cyclopts==4.6.0
+datasets==4.6.1
+detect-secrets==1.5.0
+dill==0.4.0
+diskcache==5.6.3
+distlib==0.4.0
+distro==1.9.0
+dnspython==2.8.0
+docstring-parser==0.17.0
+docutils==0.22.4
+email-validator==2.3.0
+exceptiongroup==1.3.1
+fastapi==0.135.1
+fastapi-sso==0.16.0
+fastmcp==3.1.0
+fastuuid==0.14.0
+filelock==3.25.0
+fonttools==4.61.1
+frozenlist==1.8.0
+fsspec==2026.2.0
+googleapis-common-protos==1.72.0
+griffe==1.15.0
+grpcio==1.78.0
+gunicorn==23.0.0
+h11==0.16.0
+hf-xet==1.3.2
+httpcore==1.0.9
+httptools==0.7.1
+httpx==0.28.1
+httpx-sse==0.4.3
+huggingface-hub==0.36.2
+identify==2.6.17
+idna==3.11
+ifaddr==0.2.0
+importlib-metadata==8.7.1
+iniconfig==2.3.0
+isodate==0.7.2
+itsdangerous==2.2.0
+jaraco-classes==3.4.0
+jaraco-context==6.1.0
+jaraco-functools==4.4.0
+jinja2==3.1.6
+jiter==0.13.0
+jmespath==1.1.0
+json-schema-to-pydantic==0.4.9
+jsonref==1.1.0
+jsonschema==4.26.0
+jsonschema-path==0.4.4
+jsonschema-specifications==2025.9.1
+keyring==25.7.0
+kiwisolver==1.4.9
+litellm==1.82.0
+litellm-enterprise==0.1.33
+litellm-proxy-extras==0.4.50
+markdown-it-py==4.0.0
+markdown2==2.5.5
+markupsafe==3.0.3
+matplotlib==3.10.8
+mcp==1.26.0
+mdurl==0.1.2
+more-itertools==10.8.0
+msal==1.35.0
+msal-extensions==1.3.1
+multidict==6.7.1
+multiprocess==0.70.18
+nicegui==3.8.0
+nodeenv==1.10.0
+numpy==2.4.2
+oauthlib==3.3.1
+openai==2.24.0
+openai-agents==0.10.3
+openapi-pydantic==0.5.1
+opentelemetry-api==1.39.1
+opentelemetry-exporter-otlp-proto-common==1.39.1
+opentelemetry-exporter-otlp-proto-grpc==1.39.1
+opentelemetry-exporter-otlp-proto-http==1.39.1
+opentelemetry-proto==1.39.1
+opentelemetry-sdk==1.39.1
+opentelemetry-semantic-conventions==0.60b1
+opentelemetry-semantic-conventions-ai==0.4.15
+orjson==3.11.7
+packaging==26.0
+pandas==3.0.1
+pathable==0.5.0
+patsy==1.0.2
+pillow==12.1.1
+platformdirs==4.9.2
+pluggy==1.6.0
+polars==1.38.1
+polars-runtime-32==1.38.1
+pre-commit==4.5.1
+propcache==0.4.1
+protobuf==6.33.5
+py-key-value-aio==0.4.4
+pyarrow==23.0.1
+pycparser==3.0
+pydantic==2.12.5
+pydantic-core==2.41.5
+pydantic-settings==2.13.1
+pygments==2.19.2
+pyjwt==2.11.0
+pynacl==1.6.2
+pyparsing==3.3.2
+pyperclip==1.11.0
+pyroscope-io==0.8.16
+pytest==9.0.2
+pytest-asyncio==1.3.0
+pytest-mock==3.15.1
+python-dateutil==2.9.0.post0
+python-discovery==1.1.0
+python-dotenv==1.2.2
+python-engineio==4.13.1
+python-multipart==0.0.22
+python-socketio==5.16.1
+pytz==2026.1.post1
+pyyaml==6.0.3
+redis==7.2.1
+referencing==0.37.0
+regex==2026.2.28
+requests==2.32.5
+rich==14.3.3
+rich-click==1.9.7
+rich-rst==1.3.2
+rpds-py==0.30.0
+rq==2.7.0
+ruff==0.15.4
+s3transfer==0.14.0
+scipy==1.17.1
+simple-websocket==1.1.0
+six==1.17.0
+smolagents==1.24.0
+sniffio==1.3.1
+soundfile==0.12.1
+sse-starlette==3.3.2
+starlette==0.52.1
+statsmodels==0.14.6
+tiktoken==0.12.0
+tokenizers==0.22.2
+tqdm==4.67.3
+types-requests==2.32.4.20260107
+typing-extensions==4.15.0
+typing-inspection==0.4.2
+tzlocal==5.3.1
+uncalled-for==0.2.0
+urllib3==2.6.3
+uvicorn==0.41.0
+uvloop==0.21.0
+virtualenv==21.1.0
+watchfiles==1.1.1
+websockets==15.0.1
+wikipedia-api==0.10.0
+wikipedia-mcp==2.0.0
+wsproto==1.3.2
+xxhash==3.6.0
+yarl==1.23.0
+zipp==3.23.0
diff --git a/labs/AgentStream/exgentic/misc/security/requirements/swebench/requirements.txt b/labs/AgentStream/exgentic/misc/security/requirements/swebench/requirements.txt
new file mode 100644
index 00000000..c9d1ff68
--- /dev/null
+++ b/labs/AgentStream/exgentic/misc/security/requirements/swebench/requirements.txt
@@ -0,0 +1,194 @@
+aiofiles==25.1.0
+aiohappyeyeballs==2.6.1
+aiohttp==3.13.3
+aiosignal==1.4.0
+annotated-doc==0.0.4
+annotated-types==0.7.0
+anyio==4.12.1
+apscheduler==3.11.2
+attrs==25.4.0
+azure-core==1.38.2
+azure-identity==1.25.2
+azure-storage-blob==12.28.0
+backoff==2.2.1
+beautifulsoup4==4.14.3
+bidict==0.23.1
+boto3==1.40.76
+botocore==1.40.76
+cbor2==5.8.0
+certifi==2026.2.25
+cffi==2.0.0
+cfgv==3.5.0
+chardet==6.0.0.post1
+charset-normalizer==3.4.4
+click==8.3.1
+cloudpickle==3.1.2
+codespell==2.4.1
+colorama==0.4.6
+contourpy==1.3.3
+croniter==6.0.0
+cryptography==46.0.5
+cycler==0.12.1
+datasets==4.6.1
+detect-secrets==1.5.0
+dill==0.4.0
+diskcache==5.6.3
+distlib==0.4.0
+distro==1.9.0
+dnspython==2.8.0
+docker==7.1.0
+docutils==0.22.4
+email-validator==2.3.0
+fastapi==0.135.1
+fastapi-sso==0.16.0
+fastcore==1.12.22
+fastuuid==0.14.0
+filelock==3.25.0
+fonttools==4.61.1
+frozenlist==1.8.0
+fsspec==2026.2.0
+ghapi==1.0.13
+gitdb==4.0.12
+gitpython==3.1.46
+googleapis-common-protos==1.72.0
+griffe==1.15.0
+grpcio==1.78.0
+grpclib==0.4.9
+gunicorn==23.0.0
+h11==0.16.0
+h2==4.3.0
+hf-xet==1.3.2
+hpack==4.1.0
+httpcore==1.0.9
+httptools==0.7.1
+httpx==0.28.1
+httpx-sse==0.4.3
+huggingface-hub==0.36.2
+hyperframe==6.1.0
+identify==2.6.17
+idna==3.11
+ifaddr==0.2.0
+importlib-metadata==8.7.1
+iniconfig==2.3.0
+isodate==0.7.2
+itsdangerous==2.2.0
+jinja2==3.1.6
+jiter==0.13.0
+jmespath==1.1.0
+json-schema-to-pydantic==0.4.9
+jsonschema==4.26.0
+jsonschema-specifications==2025.9.1
+kiwisolver==1.4.9
+linkify-it-py==2.1.0
+litellm==1.82.0
+litellm-enterprise==0.1.33
+litellm-proxy-extras==0.4.50
+markdown-it-py==4.0.0
+markdown2==2.5.5
+markupsafe==3.0.3
+matplotlib==3.10.8
+mcp==1.26.0
+mdit-py-plugins==0.5.0
+mdurl==0.1.2
+modal==1.3.4
+msal==1.35.0
+msal-extensions==1.3.1
+multidict==6.7.1
+multiprocess==0.70.18
+nicegui==3.8.0
+nodeenv==1.10.0
+numpy==2.4.2
+oauthlib==3.3.1
+openai==2.24.0
+openai-agents==0.10.3
+opentelemetry-api==1.39.1
+opentelemetry-exporter-otlp-proto-common==1.39.1
+opentelemetry-exporter-otlp-proto-grpc==1.39.1
+opentelemetry-exporter-otlp-proto-http==1.39.1
+opentelemetry-proto==1.39.1
+opentelemetry-sdk==1.39.1
+opentelemetry-semantic-conventions==0.60b1
+opentelemetry-semantic-conventions-ai==0.4.15
+orjson==3.11.7
+packaging==26.0
+pandas==3.0.1
+patsy==1.0.2
+pillow==12.1.1
+platformdirs==4.9.2
+pluggy==1.6.0
+polars==1.38.1
+polars-runtime-32==1.38.1
+pre-commit==4.5.1
+prompt-toolkit==3.0.52
+propcache==0.4.1
+protobuf==6.33.5
+pyarrow==23.0.1
+pycparser==3.0
+pydantic==2.12.5
+pydantic-core==2.41.5
+pydantic-settings==2.13.1
+pygments==2.19.2
+pyjwt==2.11.0
+pynacl==1.6.2
+pyparsing==3.3.2
+pyroscope-io==0.8.16
+pytest==9.0.2
+pytest-asyncio==1.3.0
+pytest-mock==3.15.1
+python-dateutil==2.9.0.post0
+python-discovery==1.1.0
+python-dotenv==1.2.2
+python-engineio==4.13.1
+python-multipart==0.0.22
+python-socketio==5.16.1
+pytz==2026.1.post1
+pyyaml==6.0.3
+redis==7.2.1
+referencing==0.37.0
+regex==2026.2.28
+requests==2.32.5
+rich==14.3.3
+rich-click==1.9.7
+rpds-py==0.30.0
+rq==2.7.0
+ruff==0.15.4
+s3transfer==0.14.0
+scipy==1.17.1
+shellingham==1.5.4
+simple-websocket==1.1.0
+six==1.17.0
+smmap==5.0.2
+smolagents==1.24.0
+sniffio==1.3.1
+soundfile==0.12.1
+soupsieve==2.8.3
+sse-starlette==3.3.2
+starlette==0.52.1
+statsmodels==0.14.6
+synchronicity==0.11.1
+tenacity==9.1.4
+textual==8.0.1
+tiktoken==0.12.0
+tokenizers==0.22.2
+toml==0.10.2
+tqdm==4.67.3
+typer==0.24.1
+types-certifi==2021.10.8.3
+types-requests==2.32.4.20260107
+types-toml==0.10.8.20240310
+typing-extensions==4.15.0
+typing-inspection==0.4.2
+tzlocal==5.3.1
+uc-micro-py==2.0.0
+unidiff==0.7.5
+urllib3==2.6.3
+uvicorn==0.41.0
+uvloop==0.21.0
+virtualenv==21.1.0
+watchfiles==1.1.1
+wcwidth==0.6.0
+websockets==15.0.1
+wsproto==1.3.2
+xxhash==3.6.0
+yarl==1.23.0
+zipp==3.23.0
diff --git a/labs/AgentStream/exgentic/misc/security/requirements/tau2/requirements.txt b/labs/AgentStream/exgentic/misc/security/requirements/tau2/requirements.txt
new file mode 100644
index 00000000..2d8a2417
--- /dev/null
+++ b/labs/AgentStream/exgentic/misc/security/requirements/tau2/requirements.txt
@@ -0,0 +1,182 @@
+addict==2.4.0
+aiofiles==25.1.0
+aiohappyeyeballs==2.6.1
+aiohttp==3.13.3
+aiosignal==1.4.0
+annotated-doc==0.0.4
+annotated-types==0.7.0
+anyio==4.12.1
+appdirs==1.4.4
+apscheduler==3.11.2
+attrs==25.4.0
+azure-core==1.38.2
+azure-identity==1.25.2
+azure-storage-blob==12.28.0
+backoff==2.2.1
+bidict==0.23.1
+boto3==1.40.76
+botocore==1.40.76
+certifi==2026.2.25
+cffi==2.0.0
+cfgv==3.5.0
+charset-normalizer==3.4.4
+click==8.3.1
+cloudpickle==3.1.2
+codespell==2.4.1
+colorama==0.4.6
+contourpy==1.3.3
+croniter==6.0.0
+cryptography==46.0.5
+cycler==0.12.1
+deepdiff==8.6.1
+detect-secrets==1.5.0
+diskcache==5.6.3
+distlib==0.4.0
+distro==1.9.0
+dnspython==2.8.0
+docstring-parser==0.17.0
+docutils==0.22.4
+email-validator==2.3.0
+fastapi==0.135.1
+fastapi-sso==0.16.0
+fastuuid==0.14.0
+filelock==3.25.0
+fonttools==4.61.1
+frozenlist==1.8.0
+fs==2.4.16
+fsspec==2026.2.0
+googleapis-common-protos==1.72.0
+griffe==1.15.0
+grpcio==1.78.0
+gunicorn==23.0.0
+h11==0.16.0
+hf-xet==1.3.2
+httpcore==1.0.9
+httptools==0.7.1
+httpx==0.28.1
+httpx-sse==0.4.3
+huggingface-hub==0.36.2
+identify==2.6.17
+idna==3.11
+ifaddr==0.2.0
+importlib-metadata==8.7.1
+iniconfig==2.3.0
+isodate==0.7.2
+itsdangerous==2.2.0
+jinja2==3.1.6
+jiter==0.13.0
+jmespath==1.1.0
+joblib==1.5.3
+json-schema-to-pydantic==0.4.9
+jsonschema==4.26.0
+jsonschema-specifications==2025.9.1
+kiwisolver==1.4.9
+langfuse==3.14.5
+litellm==1.82.0
+litellm-enterprise==0.1.33
+litellm-proxy-extras==0.4.50
+loguru==0.7.3
+markdown-it-py==4.0.0
+markdown2==2.5.5
+markupsafe==3.0.3
+matplotlib==3.10.8
+mcp==1.26.0
+mdurl==0.1.2
+msal==1.35.0
+msal-extensions==1.3.1
+multidict==6.7.1
+narwhals==2.17.0
+nicegui==3.8.0
+nodeenv==1.10.0
+numpy==2.4.2
+oauthlib==3.3.1
+openai==2.24.0
+openai-agents==0.10.3
+opentelemetry-api==1.39.1
+opentelemetry-exporter-otlp-proto-common==1.39.1
+opentelemetry-exporter-otlp-proto-grpc==1.39.1
+opentelemetry-exporter-otlp-proto-http==1.39.1
+opentelemetry-proto==1.39.1
+opentelemetry-sdk==1.39.1
+opentelemetry-semantic-conventions==0.60b1
+opentelemetry-semantic-conventions-ai==0.4.15
+orderly-set==5.5.0
+orjson==3.11.7
+packaging==25.0
+pandas==3.0.1
+patsy==1.0.2
+pillow==12.1.1
+platformdirs==4.9.2
+plotly==6.6.0
+pluggy==1.6.0
+polars==1.38.1
+polars-runtime-32==1.38.1
+pre-commit==4.5.1
+propcache==0.4.1
+protobuf==6.33.5
+psutil==7.2.2
+pycparser==3.0
+pydantic==2.12.5
+pydantic-argparse==0.10.0
+pydantic-core==2.41.5
+pydantic-settings==2.13.1
+pygments==2.19.2
+pyjwt==2.11.0
+pynacl==1.6.2
+pyparsing==3.3.2
+pyroscope-io==0.8.16
+pytest==9.0.2
+pytest-asyncio==1.3.0
+pytest-mock==3.15.1
+python-dateutil==2.9.0.post0
+python-discovery==1.1.0
+python-dotenv==1.2.2
+python-engineio==4.13.1
+python-multipart==0.0.22
+python-socketio==5.16.1
+pytz==2026.1.post1
+pyyaml==6.0.3
+redis==7.2.1
+referencing==0.37.0
+regex==2026.2.28
+requests==2.32.5
+rich==14.3.3
+rich-click==1.9.7
+rpds-py==0.30.0
+rq==2.7.0
+ruff==0.15.4
+s3transfer==0.14.0
+scikit-learn==1.8.0
+scipy==1.17.1
+seaborn==0.13.2
+setuptools==82.0.0
+simple-websocket==1.1.0
+six==1.17.0
+smolagents==1.24.0
+sniffio==1.3.1
+soundfile==0.12.1
+sse-starlette==3.3.2
+starlette==0.52.1
+statsmodels==0.14.6
+tabulate==0.9.0
+tenacity==9.1.4
+threadpoolctl==3.6.0
+tiktoken==0.12.0
+tokenizers==0.22.2
+toml==0.10.2
+tqdm==4.67.3
+types-requests==2.32.4.20260107
+typing-extensions==4.15.0
+typing-inspection==0.4.2
+tzlocal==5.3.1
+urllib3==2.6.3
+uvicorn==0.41.0
+uvloop==0.21.0
+virtualenv==21.1.0
+watchdog==6.0.0
+watchfiles==1.1.1
+websockets==15.0.1
+wrapt==1.17.3
+wsproto==1.3.2
+yarl==1.23.0
+zipp==3.23.0
diff --git a/labs/AgentStream/exgentic/misc/security/setup_environments.sh b/labs/AgentStream/exgentic/misc/security/setup_environments.sh
new file mode 100644
index 00000000..d0990814
--- /dev/null
+++ b/labs/AgentStream/exgentic/misc/security/setup_environments.sh
@@ -0,0 +1,303 @@
+#!/usr/bin/env bash
+# setup_environments.sh - Create multiple UV virtual environments for Exgentic
+#
+# Usage: ./setup_environments.sh [benchmark1 benchmark2 ...]
+#
+# This script creates:
+# 1. A 'core' environment with all optional dependencies
+# 2. Benchmark-specific environments for specified benchmarks (or all if none specified)
+# 3. Generates requirements.txt files for each environment
+#
+# Examples:
+# ./setup_environments.sh # Create all environments
+# ./setup_environments.sh gsm8k hotpotqa # Create only gsm8k and hotpotqa
+# ./setup_environments.sh core # Create only core environment
+
+set -euo pipefail
+
+# Tracking arrays for summary report
+SETUP_PASSED=()
+SETUP_FAILED=()
+
+# Get the project root (two levels up from this script in misc/security/)
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+PROJECT_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
+VENV_DIR="${PROJECT_ROOT}/.venvs"
+REQUIREMENTS_DIR="${SCRIPT_DIR}/requirements"
+
+# Colors for output
+GREEN='\033[0;32m'
+BLUE='\033[0;34m'
+YELLOW='\033[1;33m'
+RED='\033[0;31m'
+NC='\033[0m' # No Color
+
+log_info() {
+ echo -e "${BLUE}[INFO]${NC} $1"
+}
+
+log_success() {
+ echo -e "${GREEN}[SUCCESS]${NC} $1"
+}
+
+log_warning() {
+ echo -e "${YELLOW}[WARNING]${NC} $1"
+}
+
+log_error() {
+ echo -e "${RED}[ERROR]${NC} $1"
+}
+
+# Check if uv is installed
+if ! command -v uv >/dev/null 2>&1; then
+ log_error "uv is not installed. Install it with: pip install uv"
+ exit 1
+fi
+
+# Parse command line arguments
+REQUESTED_BENCHMARKS=()
+if [ $# -gt 0 ]; then
+ REQUESTED_BENCHMARKS=("$@")
+fi
+ALL_BENCHMARKS=("gsm8k" "hotpotqa" "appworld" "browsecompplus" "swebench" "tau2")
+
+# Determine which benchmarks to create
+if [ ${#REQUESTED_BENCHMARKS[@]} -eq 0 ]; then
+ # No arguments provided, create all
+ CREATE_CORE=true
+ BENCHMARKS_TO_CREATE=("${ALL_BENCHMARKS[@]}")
+ log_info "No benchmarks specified, creating all environments"
+elif [ "${REQUESTED_BENCHMARKS[0]}" = "core" ] && [ ${#REQUESTED_BENCHMARKS[@]} -eq 1 ]; then
+ # Only 'core' requested
+ CREATE_CORE=true
+ BENCHMARKS_TO_CREATE=()
+ log_info "Creating only core environment"
+else
+ # Specific benchmarks requested
+ CREATE_CORE=false
+ BENCHMARKS_TO_CREATE=()
+
+ # Check if 'core' is in the list
+ for arg in "${REQUESTED_BENCHMARKS[@]}"; do
+ if [ "$arg" = "core" ]; then
+ CREATE_CORE=true
+ else
+ # Validate benchmark name
+ if [[ " ${ALL_BENCHMARKS[@]} " =~ " ${arg} " ]]; then
+ BENCHMARKS_TO_CREATE+=("$arg")
+ else
+ log_warning "Unknown benchmark: $arg (skipping)"
+ fi
+ fi
+ done
+
+ if [ "$CREATE_CORE" = true ]; then
+ log_info "Creating core environment and benchmarks: ${BENCHMARKS_TO_CREATE[*]}"
+ else
+ log_info "Creating benchmark environments: ${BENCHMARKS_TO_CREATE[*]}"
+ fi
+fi
+
+log_info "Starting environment setup..."
+log_info "Project root: ${PROJECT_ROOT}"
+log_info "Virtual environments will be created in: ${VENV_DIR}"
+
+mkdir -p "${VENV_DIR}"
+
+# 1. Create 'core' environment with all optional dependencies
+if [ "$CREATE_CORE" = true ]; then
+ log_info "Creating 'core' environment with all optional dependencies..."
+
+ uv venv "${VENV_DIR}/core" --python 3.11
+
+ # Activate and install
+ source "${VENV_DIR}/core/bin/activate"
+
+ # Install project with all optional dependencies
+ log_info "Installing base project with all optional dependencies..."
+ uv pip install -e "${PROJECT_ROOT}[smolagents,openaimacp,otel,cli,dev]"
+
+ # Generate requirements.txt for core environment
+ log_info "Generating requirements.txt for core environment..."
+ mkdir -p "${REQUIREMENTS_DIR}/core"
+ uv pip freeze | grep -v "^-e " | grep -v " @ file://" | grep -v "github.ibm.com" | grep -v " @ git+https://" | grep -v " @ git+ssh://" > "${REQUIREMENTS_DIR}/core/requirements.txt"
+ core_pkg_count=$(wc -l < "${REQUIREMENTS_DIR}/core/requirements.txt" | tr -d ' ')
+ log_success "Saved ${core_pkg_count} packages to ${REQUIREMENTS_DIR}/core/requirements.txt"
+
+ deactivate
+ log_success "Core environment created at ${VENV_DIR}/core"
+else
+ log_info "Skipping core environment creation"
+fi
+
+# 2. Create benchmark-specific environments
+if [ ${#BENCHMARKS_TO_CREATE[@]} -gt 0 ]; then
+ BENCHMARKS=("${BENCHMARKS_TO_CREATE[@]}")
+else
+ BENCHMARKS=()
+fi
+
+if [ ${#BENCHMARKS[@]} -gt 0 ]; then
+for benchmark in "${BENCHMARKS[@]}"; do
+ log_info "Creating '${benchmark}' environment..."
+
+ SETUP_SCRIPT="${PROJECT_ROOT}/src/exgentic/benchmarks/${benchmark}/setup.sh"
+
+ if [ ! -f "${SETUP_SCRIPT}" ]; then
+ log_warning "Setup script not found: ${SETUP_SCRIPT}, skipping..."
+ continue
+ fi
+
+ # Create virtual environment
+ uv venv "${VENV_DIR}/${benchmark}" --python 3.11
+
+ # Activate environment
+ source "${VENV_DIR}/${benchmark}/bin/activate"
+
+ # Install base project with all optional dependencies
+ log_info "Installing base project with all optional dependencies for ${benchmark}..."
+ uv pip install -e "${PROJECT_ROOT}[smolagents,openaimacp,otel,cli,dev]"
+
+ # Run benchmark-specific setup script
+ log_info "Running setup script for ${benchmark}..."
+
+ # Change to project root before running setup script
+ # (some scripts expect to be run from project root)
+ cd "${PROJECT_ROOT}"
+
+ SETUP_OK=true
+ if bash "${SETUP_SCRIPT}"; then
+ log_success "${benchmark} setup completed successfully"
+ SETUP_PASSED+=("${benchmark}")
+ else
+ log_warning "${benchmark} setup script encountered issues (exit code: $?)"
+ SETUP_FAILED+=("${benchmark}")
+ SETUP_OK=false
+ fi
+
+ # Generate requirements.txt only when setup succeeded
+ if [ "${SETUP_OK}" = true ]; then
+ log_info "Generating requirements.txt for ${benchmark} environment..."
+ mkdir -p "${REQUIREMENTS_DIR}/${benchmark}"
+ uv pip freeze | grep -v "^-e " | grep -v " @ file://" | grep -v "github.ibm.com" | grep -v " @ git+https://" | grep -v " @ git+ssh://" > "${REQUIREMENTS_DIR}/${benchmark}/requirements.txt"
+ bench_pkg_count=$(wc -l < "${REQUIREMENTS_DIR}/${benchmark}/requirements.txt" | tr -d ' ')
+ log_success "Saved ${bench_pkg_count} packages to ${REQUIREMENTS_DIR}/${benchmark}/requirements.txt"
+ else
+ log_warning "Skipping requirements.txt generation for ${benchmark} due to setup failure"
+ fi
+
+ deactivate
+ log_success "${benchmark} environment created at ${VENV_DIR}/${benchmark}"
+done
+fi
+
+# Generate README for requirements
+log_info "Generating requirements README..."
+cat > "${REQUIREMENTS_DIR}/README.md" << 'EOF'
+# Requirements Files
+
+This directory contains frozen requirements for each virtual environment.
+
+## Directories
+
+Each environment has its own directory with a `requirements.txt` file:
+
+- `core/requirements.txt` - Core environment with all optional dependencies
+- `gsm8k/requirements.txt` - GSM8K benchmark environment
+- `hotpotqa/requirements.txt` - HotpotQA benchmark environment
+- `appworld/requirements.txt` - AppWorld benchmark environment
+- `browsecompplus/requirements.txt` - BrowseComp+ benchmark environment
+- `swebench/requirements.txt` - SWE-bench benchmark environment
+- `tau2/requirements.txt` - TAU-2 benchmark environment
+
+## Usage
+
+To recreate an environment from a requirements file:
+
+```bash
+# Create a new virtual environment
+uv venv .venv --python 3.11
+
+# Activate it
+source .venv/bin/activate
+
+# Install from requirements
+uv pip install -r misc/security/requirements/core/requirements.txt
+```
+
+## Notes
+
+- These files are generated automatically by `misc/security/setup_environments.sh`
+- They represent the exact package versions installed in each environment
+- Only PyPI packages are included (no git or local installs)
+- Regenerate by re-running the setup script
+
+## Generation Date
+
+EOF
+
+log_success "Requirements README created at ${REQUIREMENTS_DIR}/README.md"
+
+# Summary
+echo ""
+log_success "Environment setup completed!"
+echo ""
+echo "Created environments:"
+if [ "$CREATE_CORE" = true ] && [ -d "${VENV_DIR}/core" ]; then
+ echo " ${VENV_DIR}/core - Core environment with all optional dependencies"
+ if [ -f "${REQUIREMENTS_DIR}/core/requirements.txt" ]; then
+ pkg_count=$(wc -l < "${REQUIREMENTS_DIR}/core/requirements.txt" | tr -d ' ')
+ echo " Requirements: ${REQUIREMENTS_DIR}/core/requirements.txt (${pkg_count} packages)"
+ fi
+fi
+if [ ${#BENCHMARKS[@]} -gt 0 ]; then
+for benchmark in "${BENCHMARKS[@]}"; do
+ if [ -d "${VENV_DIR}/${benchmark}" ]; then
+ echo " ${VENV_DIR}/${benchmark} - ${benchmark} benchmark environment"
+ if [ -f "${REQUIREMENTS_DIR}/${benchmark}/requirements.txt" ]; then
+ pkg_count=$(wc -l < "${REQUIREMENTS_DIR}/${benchmark}/requirements.txt" | tr -d ' ')
+ echo " Requirements: ${REQUIREMENTS_DIR}/${benchmark}/requirements.txt (${pkg_count} packages)"
+ fi
+ fi
+done
+fi
+echo ""
+echo "To activate an environment, use:"
+if [ "$CREATE_CORE" = true ]; then
+ echo " source ${VENV_DIR}/core/bin/activate"
+fi
+if [ ${#BENCHMARKS[@]} -gt 0 ]; then
+for benchmark in "${BENCHMARKS[@]}"; do
+ if [ -d "${VENV_DIR}/${benchmark}" ]; then
+ echo " source ${VENV_DIR}/${benchmark}/bin/activate"
+ fi
+done
+fi
+echo ""
+echo "Note: Some benchmarks may require additional prerequisites:"
+echo " - appworld, swebench: Git LFS"
+echo " - browsecompplus: Java 21+, SSH access to IBM GitHub"
+echo ""
+echo "Usage examples:"
+echo " ./misc/security/setup_environments.sh # Create all environments"
+echo " ./misc/security/setup_environments.sh gsm8k hotpotqa # Create specific benchmarks"
+echo " ./misc/security/setup_environments.sh core # Create only core environment"
+
+# Setup script pass/fail summary
+echo ""
+echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+echo " Setup Script Results"
+echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
+if [ ${#SETUP_PASSED[@]} -gt 0 ]; then
+ for b in "${SETUP_PASSED[@]}"; do
+ echo -e " ${GREEN}✔ PASSED${NC} ${b}"
+ done
+else
+ echo " (no benchmark setup scripts ran)"
+fi
+if [ ${#SETUP_FAILED[@]} -gt 0 ]; then
+ for b in "${SETUP_FAILED[@]}"; do
+ echo -e " ${RED}✘ FAILED${NC} ${b}"
+ done
+fi
+echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
diff --git a/labs/AgentStream/exgentic/misc/skills/add-agent/SKILL.md b/labs/AgentStream/exgentic/misc/skills/add-agent/SKILL.md
new file mode 100644
index 00000000..047d8a16
--- /dev/null
+++ b/labs/AgentStream/exgentic/misc/skills/add-agent/SKILL.md
@@ -0,0 +1,68 @@
+---
+name: add-agent
+description: Use when adding or updating an agent adapter in the Exgentic repository. Follow the repository agent principles, separate lightweight config from heavy execution logic, isolate third-party dependencies behind lazy imports, adapt to any benchmark contract without requiring benchmark changes, and validate the adapter with representative smoke tests before finishing.
+---
+
+# Add Agent
+
+Use this skill when working on agent adapters in the Exgentic repository.
+
+## First read
+
+Start with:
+- `docs/adding-agents.md`
+- `src/exgentic/core/agent.py`
+- `src/exgentic/core/agent_instance.py`
+- `src/exgentic/interfaces/registry.py`
+
+Then inspect the most relevant existing adapters:
+- `src/exgentic/agents/litellm_tool_calling/litellm_tool_calling_agent.py` + `instance.py` (split pattern: heavy deps in separate file)
+- `src/exgentic/agents/cli/claude/agent.py` (light pattern: everything in one file)
+
+## Workflow
+
+1. Decide whether to split files.
+ If the agent depends on heavy third-party libraries (litellm, smolagents, openai SDK, etc.), put the Agent in one file and the AgentInstance in `instance.py`. If deps are light, keep both in a single file.
+
+2. Implement the Agent class first.
+ Subclass `Agent`. Declare `display_name` and `slug_name` as `ClassVar[str]`. Add user-facing config fields. Implement `_get_instance_class()` with a lazy import. Implement `_get_instance_kwargs()` to translate config and the benchmark contract into instance constructor kwargs.
+
+3. Implement the AgentInstance class.
+ Subclass `AgentInstance`. Accept `session_id` plus the kwargs from `_get_instance_kwargs()`. Call `super().__init__(session_id)`. Implement `react()` as the core decision loop and `close()` for cleanup. Optionally override `start()` and `get_cost()`.
+
+4. Add `requirements.txt` for agent-specific dependencies.
+ List only packages not already in the base exgentic install. Place the file in the agent's package directory; `RunnerMixin` discovers it automatically.
+
+5. Add `setup.sh` if non-pip setup is needed.
+ Place it next to the agent module; `RunnerMixin` discovers it automatically.
+
+6. Register the agent in `src/exgentic/interfaces/registry.py`.
+ Add a `RegistryEntry` to the `AGENTS` dict. Ensure `slug_name` and `display_name` match the class exactly.
+
+7. Validate the adapter as an agent, not just as code.
+ Check registry loading, dependency isolation, at least one end-to-end benchmark run, cost reporting, and cleanup.
+
+## Non-negotiable rules
+
+- The Agent file must be importable without installing agent-specific packages.
+- `_get_instance_class()` must use a lazy import to isolate heavy deps.
+- `_get_instance_kwargs()` must faithfully pass the benchmark contract (task, context, actions, session_id) through to the instance.
+- The instance constructor must call `super().__init__(session_id)`.
+- `react()` must return `None` when the agent decides it is done.
+- `close()` must not raise exceptions.
+- `slug_name` and `display_name` in the registry entry must exactly match the class values.
+- The agent must adapt to the benchmark, never the other way around.
+
+## Validation
+
+Before finishing, run at least:
+- `python -m py_compile` on changed agent files
+- `pre-commit run --files ...`
+- `git diff --check`
+
+Also confirm:
+- `load_agent("slug_name")` succeeds from a Python shell
+- The Agent file imports cleanly without agent-specific packages installed
+- At least one benchmark runs end to end with the new agent
+- `close()` completes without error
+- `get_cost()` returns a valid report
diff --git a/labs/AgentStream/exgentic/misc/skills/add-benchmark/SKILL.md b/labs/AgentStream/exgentic/misc/skills/add-benchmark/SKILL.md
new file mode 100644
index 00000000..3e3da2f1
--- /dev/null
+++ b/labs/AgentStream/exgentic/misc/skills/add-benchmark/SKILL.md
@@ -0,0 +1,62 @@
+---
+name: add-benchmark
+description: Use when adding or updating a benchmark adapter in the Exgentic repository. Follow the repository benchmark principles, keep the benchmark contract protocol-agnostic, prefer the thinnest possible wrapper that makes the benchmark accessible to any Exgentic agent, reuse external harness assets and scoring where possible, and validate the adapter with representative smoke tests before finishing.
+---
+
+# Add Benchmark
+
+Use this skill when working on benchmark adapters in the Exgentic repository.
+
+## First read
+
+Start with:
+- `docs/adding-benchmarks.md`
+- `src/exgentic/core/benchmark.py`
+- `src/exgentic/interfaces/registry.py`
+
+Then inspect the most relevant existing adapters:
+- `src/exgentic/benchmarks/tau2/tau2_benchmark.py`
+- `src/exgentic/benchmarks/bfcl/bfcl_benchmark.py`
+
+## Workflow
+
+1. Define the benchmark contract before writing code.
+ Decide the real `task`, the agent-relevant `context`, the semantic `actions`, the finish condition, and the scoring boundary.
+
+2. Keep the agent-facing contract protocol-agnostic.
+ Do not define the benchmark in terms of one provider's chat or tool-calling format.
+
+3. Prefer the thinnest possible wrapper.
+ Make the benchmark accessible to any Exgentic agent with the minimum translation surface necessary. Do not add extra abstraction, copied logic, or runtime machinery unless it is needed to preserve benchmark meaning.
+
+4. Decide the source-of-truth boundary.
+ Reuse external benchmark assets, setup, and scoring where possible, but do not let an external harness dictate the wrong runtime contract for Exgentic.
+
+5. Implement runtime, setup, and registration separately.
+ Prefer a benchmark module, a `setup.sh`, and a registry entry with clear responsibilities.
+
+6. Validate the adapter as a benchmark, not just as code.
+ Check task listing, subset listing, happy-path scoring, failure-path scoring, and error semantics.
+
+## Non-negotiable rules
+
+- `task` must be the actual task.
+- `context` must include only what the agent should know.
+- subset names and internal metadata stay out of `context`.
+- Prefer the thinnest wrapper that preserves the benchmark's meaning.
+- Actions should describe semantic operations, not protocol artifacts.
+- Use `finish` only when it is part of the benchmark contract.
+- If outputs are not real execution results, say so plainly in the benchmark contract.
+- Keep success, unsuccessful completion, unfinished runs, and errors distinct.
+
+## Validation
+
+Before finishing, run at least:
+- `python -m py_compile` on changed benchmark files
+- `pre-commit run --files ...`
+- `git diff --check`
+
+Also run benchmark-specific smoke tests that prove:
+- one passing case works
+- one failing case is represented correctly
+- one real error is surfaced as an error
diff --git a/labs/AgentStream/exgentic/misc/utils/.secrets.baseline b/labs/AgentStream/exgentic/misc/utils/.secrets.baseline
new file mode 100644
index 00000000..3e5ec796
--- /dev/null
+++ b/labs/AgentStream/exgentic/misc/utils/.secrets.baseline
@@ -0,0 +1,150 @@
+{
+ "version": "1.5.0",
+ "plugins_used": [
+ {
+ "name": "ArtifactoryDetector"
+ },
+ {
+ "name": "AWSKeyDetector"
+ },
+ {
+ "name": "AzureStorageKeyDetector"
+ },
+ {
+ "name": "Base64HighEntropyString",
+ "limit": 4.5
+ },
+ {
+ "name": "BasicAuthDetector"
+ },
+ {
+ "name": "CloudantDetector"
+ },
+ {
+ "name": "DiscordBotTokenDetector"
+ },
+ {
+ "name": "GitHubTokenDetector"
+ },
+ {
+ "name": "GitLabTokenDetector"
+ },
+ {
+ "name": "HexHighEntropyString",
+ "limit": 3.0
+ },
+ {
+ "name": "IbmCloudIamDetector"
+ },
+ {
+ "name": "IbmCosHmacDetector"
+ },
+ {
+ "name": "IPPublicDetector"
+ },
+ {
+ "name": "JwtTokenDetector"
+ },
+ {
+ "name": "KeywordDetector",
+ "keyword_exclude": ""
+ },
+ {
+ "name": "MailchimpDetector"
+ },
+ {
+ "name": "NpmDetector"
+ },
+ {
+ "name": "OpenAIDetector"
+ },
+ {
+ "name": "PrivateKeyDetector"
+ },
+ {
+ "name": "PypiTokenDetector"
+ },
+ {
+ "name": "SendGridDetector"
+ },
+ {
+ "name": "SlackDetector"
+ },
+ {
+ "name": "SoftlayerDetector"
+ },
+ {
+ "name": "SquareOAuthDetector"
+ },
+ {
+ "name": "StripeDetector"
+ },
+ {
+ "name": "TelegramBotTokenDetector"
+ },
+ {
+ "name": "TwilioKeyDetector"
+ }
+ ],
+ "filters_used": [
+ {
+ "path": "detect_secrets.filters.allowlist.is_line_allowlisted"
+ },
+ {
+ "path": "detect_secrets.filters.common.is_baseline_file",
+ "filename": "labs/AgentStream/exgentic/misc/utils/.secrets.baseline"
+ },
+ {
+ "path": "detect_secrets.filters.common.is_ignored_due_to_verification_policies",
+ "min_level": 2
+ },
+ {
+ "path": "detect_secrets.filters.heuristic.is_indirect_reference"
+ },
+ {
+ "path": "detect_secrets.filters.heuristic.is_likely_id_string"
+ },
+ {
+ "path": "detect_secrets.filters.heuristic.is_lock_file"
+ },
+ {
+ "path": "detect_secrets.filters.heuristic.is_not_alphanumeric_string"
+ },
+ {
+ "path": "detect_secrets.filters.heuristic.is_potential_uuid"
+ },
+ {
+ "path": "detect_secrets.filters.heuristic.is_prefixed_with_dollar_sign"
+ },
+ {
+ "path": "detect_secrets.filters.heuristic.is_sequential_string"
+ },
+ {
+ "path": "detect_secrets.filters.heuristic.is_swagger_file"
+ },
+ {
+ "path": "detect_secrets.filters.heuristic.is_templated_secret"
+ }
+ ],
+ "results": {
+ "labs/AgentStream/exgentic/tests/benchmarks/recordings/appworld/trajectory.jsonl": [
+ {
+ "type": "JSON Web Token",
+ "filename": "labs/AgentStream/exgentic/tests/benchmarks/recordings/appworld/trajectory.jsonl",
+ "hashed_secret": "0c677ddce015761585645772702fd153efbb08d9",
+ "is_verified": false,
+ "line_number": 7
+ }
+ ],
+ "labs/AgentStream/exgentic/tests/benchmarks/recordings/swebench/results.json": [
+ {
+ "type": "Secret Keyword",
+ "filename": "labs/AgentStream/exgentic/tests/benchmarks/recordings/swebench/results.json",
+ "hashed_secret": "d4e0e04792fd434b5dc9c4155c178f66edcf4ed3",
+ "is_verified": false,
+ "line_number": 46
+ }
+ ]
+ },
+ "generated_at": "2026-03-17T11:41:59Z"
+}
diff --git a/labs/AgentStream/exgentic/misc/utils/enforce_dependency_caps.py b/labs/AgentStream/exgentic/misc/utils/enforce_dependency_caps.py
new file mode 100644
index 00000000..e2e07a01
--- /dev/null
+++ b/labs/AgentStream/exgentic/misc/utils/enforce_dependency_caps.py
@@ -0,0 +1,124 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+#!/usr/bin/env python3
+"""Enforce that all direct dependencies in pyproject.toml have upper version bounds.
+
+This script prevents supply chain attacks by ensuring no dependency can auto-upgrade
+to an arbitrary future version. All dependencies must be capped at the next major
+version (e.g., >=1.0.0,<2).
+
+Exit codes:
+ 0: All dependencies have upper bounds
+ 1: One or more dependencies lack upper bounds
+"""
+
+import re
+import sys
+from pathlib import Path
+
+
+def _extract_dependency_lines(content: str) -> list[str]:
+ """Extract lines that belong to dependency sections in pyproject.toml.
+
+ Scopes extraction to [project.dependencies] and
+ [project.optional-dependencies.*] sections only, so that version-like
+ strings in other sections (e.g. build-system.requires) are ignored.
+ """
+ lines: list[str] = []
+ in_dep_section = False
+ in_dep_array = False
+
+ for line in content.splitlines():
+ stripped = line.strip()
+
+ # Detect section headers
+ if stripped.startswith("["):
+ in_dep_section = stripped in ("[project]",) or stripped.startswith("[project.optional-dependencies")
+ in_dep_array = False
+ continue
+
+ if not in_dep_section:
+ continue
+
+ # Inside a relevant section, look for dependency array starts
+ if "dependencies" in stripped and "=" in stripped and "[" in stripped:
+ in_dep_array = True
+ continue
+ # Also handle bare list continuation under optional-dependencies groups
+ if stripped.startswith('"') and in_dep_section and not in_dep_array:
+ # We're likely in an optional-dep group list
+ in_dep_array = True
+
+ if in_dep_array:
+ if stripped == "]":
+ in_dep_array = False
+ continue
+ lines.append(line)
+
+ return lines
+
+
+def check_dependency_caps(pyproject_path: Path) -> list[str]:
+ """Check all dependencies in pyproject.toml for upper version bounds.
+
+ Args:
+ pyproject_path: Path to pyproject.toml file
+
+ Returns:
+ List of dependency lines that lack upper bounds (empty if all are capped)
+ """
+ content = pyproject_path.read_text()
+ uncapped = []
+
+ # Only check lines inside dependency sections
+ dep_lines = _extract_dependency_lines(content)
+
+ # Pattern to match dependency specifications (supports extras like [extra])
+ # Matches: "package>=1.0.0" or "package[extra]>=1.0.0,!=1.2.3" but not "package>=1.0.0,<2"
+ dep_pattern = re.compile(
+ r'^\s*"([a-zA-Z0-9_-]+(?:\[[a-zA-Z0-9_,\s-]+\])?)([><=!,.\d\s]+)"',
+ )
+
+ for line in dep_lines:
+ match = dep_pattern.match(line)
+ if match:
+ full_line = match.group(0).strip()
+ version_spec = match.group(2)
+
+ # Check if there's an upper bound (< or <=)
+ if "<" not in version_spec:
+ uncapped.append(full_line)
+
+ return uncapped
+
+
+def main() -> int:
+ """Main entry point."""
+ project_root = Path(__file__).resolve().parents[2]
+ pyproject_path = project_root / "pyproject.toml"
+
+ if not pyproject_path.exists():
+ print("Error: pyproject.toml not found", file=sys.stderr)
+ return 1
+
+ uncapped = check_dependency_caps(pyproject_path)
+
+ if uncapped:
+ print("ERROR: Dependencies without upper version bounds found:", file=sys.stderr)
+ print(file=sys.stderr)
+ for dep in uncapped:
+ print(f" {dep}", file=sys.stderr)
+ print(file=sys.stderr)
+ print(
+ "All dependencies must have upper bounds (e.g., >=1.0.0,<2) to limit supply chain attack exposure.",
+ file=sys.stderr,
+ )
+ return 1
+
+ print("All dependencies have upper version bounds")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/labs/AgentStream/exgentic/misc/utils/enforce_library_imports.py b/labs/AgentStream/exgentic/misc/utils/enforce_library_imports.py
new file mode 100644
index 00000000..6f1058d9
--- /dev/null
+++ b/labs/AgentStream/exgentic/misc/utils/enforce_library_imports.py
@@ -0,0 +1,32 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+import logging
+import sys
+
+logging.basicConfig(level=logging.INFO, format="%(message)s")
+logger = logging.getLogger(__name__)
+
+
+def check_imports(file_path):
+ success = True
+ with open(file_path, encoding="utf-8") as file:
+ for lineno, line in enumerate(file, start=1):
+ if "from src import" in line or "from src." in line or "import src" in line:
+ logger.error(f"Non library import: {file_path}:{lineno}: {line.strip()[:30]}...")
+ success = False
+ return success
+
+
+def main():
+ success = True
+ for file_path in sys.argv[1:]:
+ if not check_imports(file_path):
+ success = False
+
+ if not success:
+ sys.exit(1)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/labs/AgentStream/exgentic/misc/utils/enforce_relative_imports.py b/labs/AgentStream/exgentic/misc/utils/enforce_relative_imports.py
new file mode 100644
index 00000000..8531541e
--- /dev/null
+++ b/labs/AgentStream/exgentic/misc/utils/enforce_relative_imports.py
@@ -0,0 +1,41 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+import logging
+import sys
+
+logging.basicConfig(level=logging.INFO, format="%(message)s")
+logger = logging.getLogger(__name__)
+
+library_name = "exgentic"
+
+
+def check_imports(file_path):
+ success = True
+ with open(file_path, encoding="utf-8") as file:
+ for lineno, line in enumerate(file, start=1):
+ if (
+ f"from {library_name} import" in line
+ or f"from {library_name}." in line
+ or f"import {library_name}" in line
+ or "from src import" in line
+ or "from src." in line
+ or "import src" in line
+ ):
+ logger.error(f"Non relative import: {file_path}:{lineno}: {line.strip()[:30]}...")
+ success = False
+ return success
+
+
+def main():
+ success = True
+ for file_path in sys.argv[1:]:
+ if not check_imports(file_path):
+ success = False
+
+ if not success:
+ sys.exit(1)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/labs/AgentStream/exgentic/misc/utils/enforce_spdx_header.py b/labs/AgentStream/exgentic/misc/utils/enforce_spdx_header.py
new file mode 100644
index 00000000..51378aeb
--- /dev/null
+++ b/labs/AgentStream/exgentic/misc/utils/enforce_spdx_header.py
@@ -0,0 +1,71 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+#!/usr/bin/env python3
+"""Ensure SPDX+copyright header exists at top of Python files."""
+
+from __future__ import annotations
+
+import re
+import sys
+from pathlib import Path
+
+HEADER_LINES = [
+ "# SPDX-License-Identifier: Apache-2.0",
+ "# Copyright (C) 2026, The AgentStream organization and its contributors.",
+]
+HEADER_TEXT = "\n".join(HEADER_LINES) + "\n\n"
+
+COPYRIGHT_RE = re.compile(
+ r"^# Copyright \(C\) \d{4}, The (?:Exgentic|AgentStream) organization and its contributors\.$"
+)
+APACHE_LICENSE_MARKER = 'Licensed under the Apache License, Version 2.0 (the "License")'
+
+SKIP_DIRS = {
+ ".git",
+ ".venv",
+ ".pytest_cache",
+ ".ruff_cache",
+ "__pycache__",
+}
+
+
+def should_skip(path: Path) -> bool:
+ return bool(set(path.parts) & SKIP_DIRS)
+
+
+def update_file(path: Path) -> bool:
+ raw = path.read_bytes()
+ original = raw.decode("utf-8", errors="surrogateescape")
+ if not original:
+ return False
+
+ lines = original.splitlines()
+ if len(lines) >= 2 and lines[0] == HEADER_LINES[0] and COPYRIGHT_RE.match(lines[1]):
+ return False
+
+ # Preserve complete Apache-2.0 notices carried by third-party source files.
+ if APACHE_LICENSE_MARKER in "\n".join(lines[:20]):
+ return False
+
+ updated = HEADER_TEXT + original
+ path.write_text(updated, encoding="utf-8", errors="surrogateescape")
+ return True
+
+
+def main(argv: list[str]) -> int:
+ changed = False
+ for filename in argv:
+ path = Path(filename)
+ if not path.is_file() or should_skip(path):
+ continue
+ if update_file(path):
+ changed = True
+ if changed:
+ print("SPDX headers updated. Re-run pre-commit.")
+ return 1
+ return 0
+
+
+if __name__ == "__main__":
+ raise SystemExit(main(sys.argv[1:]))
diff --git a/labs/AgentStream/exgentic/pyproject.toml b/labs/AgentStream/exgentic/pyproject.toml
new file mode 100644
index 00000000..af133ae1
--- /dev/null
+++ b/labs/AgentStream/exgentic/pyproject.toml
@@ -0,0 +1,104 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "exgentic"
+# Bundled Exgentic snapshot used by AgentStream; not published independently from Sico.
+version = "0.0.0+agentstream"
+description = "Exgentic - General agent evaluation"
+authors = [{name = "Exgentic Team"}]
+license = {text = "Apache-2.0"}
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = [
+ "typing-extensions>=4,<5",
+ "pydantic-settings>=2,<3",
+ "nicegui>=3,<4",
+ "cloudpickle>=3,<4",
+ "diskcache>=5,<6",
+ "filelock>=3,<4",
+ "click>=8.1.7,<9",
+ "json-schema-to-pydantic>=0.4,<1",
+ "litellm>=1.65.0,!=1.82.7,!=1.82.8,<2",
+ "mcp>=1.24,<2",
+ "pydantic>=2.9.2,<3",
+ "python-dotenv>=1,<2",
+ "rich>=13,<14",
+ "rich-click>=1,<2",
+]
+
+[project.scripts]
+exgentic = "exgentic.interfaces.cli.main:main"
+
+[project.optional-dependencies]
+# Observability
+otel = [
+ "opentelemetry-api>=1,<2",
+ "opentelemetry-sdk>=1,<2",
+ "opentelemetry-exporter-otlp-proto-http>=1,<2",
+ "opentelemetry-exporter-otlp-proto-grpc>=1,<2",
+ "opentelemetry-semantic-conventions-ai>=0.4.0,<1",
+]
+analysis = [
+ "matplotlib>=3,<4",
+ "numpy>=2,<3",
+ "pandas>=3,<4",
+ "scipy>=1,<2",
+ "statsmodels>=0.14,<1",
+]
+amem = [
+ "sentence-transformers>=3,<5",
+ "scikit-learn>=1,<2",
+]
+
+# Development
+dev = [
+ "pytest>=7.0.0,<10",
+ "pytest-asyncio>=0.21.0,<2",
+ "pytest-mock>=3.0.0,<4",
+ "pre-commit>=3.0.0,<5",
+ "ruff>=0.1.0,<1",
+ "codespell>=2.0.0,<3",
+ "detect-secrets>=1.0.0,<2",
+]
+
+[tool.hatch.metadata]
+allow-direct-references = true
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/exgentic"]
+include = [
+ "src/exgentic/benchmarks/**/requirements.txt",
+ "src/exgentic/benchmarks/**/setup.sh",
+ "src/exgentic/benchmarks/**/system-deps.txt",
+ "src/exgentic/agents/**/requirements.txt",
+ "src/exgentic/agents/**/setup.sh",
+ "src/exgentic/agents/**/system-deps.txt",
+]
+
+[tool.hatch.build.targets.sdist]
+include = [
+ "/src",
+ "/tests",
+ "/docs",
+ "/examples",
+]
+
+[tool.uv]
+override-dependencies = [
+ "rich>=13.9.4,<14",
+]
+
+[tool.codespell]
+skip = "tests/benchmarks/recordings/*,uv.lock"
+ignore-words-list = "requestor"
+
+[tool.pytest.ini_options]
+asyncio_mode = "strict"
+pythonpath = ["."]
+
+[dependency-groups]
+dev = [
+ "pytest>=9.0.1,<10",
+]
diff --git a/labs/AgentStream/exgentic/ruff.toml b/labs/AgentStream/exgentic/ruff.toml
new file mode 100644
index 00000000..aba4f270
--- /dev/null
+++ b/labs/AgentStream/exgentic/ruff.toml
@@ -0,0 +1,78 @@
+# Ruff configuration for Exgentic project
+
+# Set line length
+line-length = 120
+
+# Exclude common directories
+extend-exclude = [
+ ".venv",
+ ".venvs",
+ "venv",
+ ".git",
+ "__pycache__",
+ ".pytest_cache",
+ ".exgentic",
+]
+
+[lint]
+# Enable specific rule sets
+select = [
+ "E", # pycodestyle errors
+ "W", # pycodestyle warnings
+ "F", # pyflakes
+ "I", # isort
+ "B", # flake8-bugbear
+ "C4", # flake8-comprehensions
+ "UP", # pyupgrade
+ "D", # pydocstyle
+ "RUF", # Ruff-specific rules
+ "RET", # flake8-return
+ "C90", # mccabe complexity
+ "N", # pep8-naming
+ "G", # flake8-logging-format
+]
+
+# Ignore specific rules
+ignore = [
+ "TID252", # Relative imports from parent modules - conflicts with our enforce-relative-imports hook
+ "C901", # Function is too complex - will be addressed separately
+ "T201", # Print statements - will be addressed separately
+ "D100", # Missing docstring in public module
+ "D101", # Missing docstring in public class
+ "D102", # Missing docstring in public method
+ "D103", # Missing docstring in public function
+ "D104", # Missing docstring in public package
+ "D105", # Missing docstring in magic method
+ "D107", # Missing docstring in __init__
+ "G004", # Logging statement uses f-string
+ "UP007", # Use `X | Y` for type annotations
+ "RET504", # Unnecessary assignment to ... before return
+]
+
+# Allow autofix for all enabled rules
+fixable = ["ALL"]
+unfixable = []
+
+[lint.per-file-ignores]
+# Allow print statements in test files and scripts
+"tests/**/*.py" = ["T201"]
+"examples/**/*.py" = ["T201"]
+"misc/**/*.py" = ["T201"]
+
+[lint.pydocstyle]
+# Use Google-style docstrings
+convention = "google"
+
+[lint.mccabe]
+# Set maximum complexity
+max-complexity = 15
+
+[format]
+# Use double quotes for strings
+quote-style = "double"
+
+# Indent with 4 spaces
+indent-style = "space"
+
+# Use Unix line endings
+line-ending = "lf"
diff --git a/labs/AgentStream/exgentic/scripts/a_mem/run_experiment.py b/labs/AgentStream/exgentic/scripts/a_mem/run_experiment.py
new file mode 100644
index 00000000..4b319465
--- /dev/null
+++ b/labs/AgentStream/exgentic/scripts/a_mem/run_experiment.py
@@ -0,0 +1,416 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from collections import defaultdict
+from datetime import datetime
+from pathlib import Path
+from typing import Any
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2] / "src"))
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "utils"))
+
+os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True"
+
+from exgentic.interfaces.lib.api import evaluate
+from exgentic.interfaces.registry import load_agent, load_benchmark
+from exgentic.agents.a_mem.memory_store import MemoryStore
+from exgentic.core.types import ModelSettings
+
+from task_ordering import get_unified_task_order, group_by_benchmark
+
+
+BENCHMARK_REGISTRY: dict[str, dict[str, Any]] = {
+ "browsecompplus": {
+ "bm_kwargs": {
+ "searcher_type": "faiss",
+ "include_get_document": True,
+ "eval_model_id": "openai/gpt-5.4",
+ },
+ "agent_kwargs": {},
+ },
+ "swebench": {
+ "bm_kwargs": {
+ "subset": "princeton-nlp/SWE-bench_Verified",
+ },
+ "agent_kwargs": {},
+ },
+ "appworld": {
+ "bm_kwargs": {
+ "subset": "test_challenge",
+ },
+ "agent_kwargs": {
+ "enable_tool_shortlisting": True,
+ "max_selected_tools": 30,
+ },
+ },
+ "bfcl": {
+ "bm_kwargs": {
+ "subset": "multi_turn_base",
+ },
+ "agent_kwargs": {},
+ },
+ "tau2": {
+ "bm_kwargs": {
+ "subset": "telecom",
+ "user_simulator_model": "openai/gpt-5.4",
+ },
+ "agent_kwargs": {},
+ },
+ "hle": {
+ "bm_kwargs": {
+ "judge_model": "openai/gpt-5.4",
+ "runner": "direct",
+ },
+ "agent_kwargs": {},
+ },
+}
+
+
+def extract_token_counts(cost_reports: dict) -> tuple[int, int]:
+ total_in, total_out = 0, 0
+ for report in cost_reports.values():
+ if isinstance(report, dict):
+ total_in += report.get("input_tokens", 0)
+ total_out += report.get("output_tokens", 0)
+ elif hasattr(report, "input_tokens"):
+ total_in += report.input_tokens
+ total_out += report.output_tokens
+ return total_in, total_out
+
+
+def get_memory_stats(mode: str, bm_slug: str) -> tuple[int, int]:
+ stores = MemoryStore.list_stores()
+ if mode == "isolated":
+ store = stores.get(f"amem_isolated_{bm_slug}")
+ elif mode == "sequential":
+ store = stores.get("amem_sequential_global")
+ elif mode == "interleaved":
+ store = stores.get("amem_interleaved_global")
+ else:
+ return 0, 0
+ if store is None:
+ return 0, 0
+ stats = store.get_stats()
+ return stats["total_memories"], stats["total_evolutions"]
+
+
+def record_online_metrics(
+ metrics_path: Path,
+ session_index: int,
+ bm_slug: str,
+ task_id: str,
+ sr: Any,
+ mode: str,
+ seed: int,
+ model: str,
+ all_scores: list[float],
+ bm_scores: dict[str, list[float]],
+):
+ score = sr.score if sr.score is not None else (1.0 if sr.success else 0.0)
+ all_scores.append(score)
+ bm_scores[bm_slug].append(score)
+
+ input_tokens, output_tokens = extract_token_counts(sr.cost_reports)
+ memory_count, total_evolutions = get_memory_stats(mode, bm_slug)
+
+ record = {
+ "session_index": session_index,
+ "seed": seed,
+ "mode": mode,
+ "agent": "a_mem",
+ "model": model,
+ "benchmark_slug": bm_slug,
+ "task_id": task_id,
+ "score": score,
+ "cumulative_avg_score": sum(all_scores) / len(all_scores),
+ "benchmark_cumulative_avg_score": (
+ sum(bm_scores[bm_slug]) / len(bm_scores[bm_slug])
+ ),
+ "steps": sr.steps,
+ "action_count": sr.action_count,
+ "agent_cost": sr.agent_cost,
+ "input_tokens": input_tokens,
+ "output_tokens": output_tokens,
+ "memory_count": memory_count,
+ "total_evolutions": total_evolutions,
+ "execution_time": sr.execution_time,
+ "status": sr.status.value if hasattr(sr.status, "value") else str(sr.status),
+ "timestamp": datetime.now().isoformat(),
+ }
+
+ with open(metrics_path, "a", encoding="utf-8") as f:
+ f.write(json.dumps(record, ensure_ascii=False) + "\n")
+
+ return record
+
+
+
+def run_experiment(args):
+ benchmarks_to_run = [s.strip() for s in args.benchmarks.split(",")]
+ configs = {k: BENCHMARK_REGISTRY[k] for k in benchmarks_to_run}
+
+ settings_kwargs = {}
+ if args.max_tokens is not None:
+ settings_kwargs["max_tokens"] = args.max_tokens
+ if args.reasoning_effort is not None:
+ settings_kwargs["reasoning_effort"] = args.reasoning_effort
+ model_settings = ModelSettings(**settings_kwargs)
+
+ print(f"\n{'=' * 70}")
+ print(f" A-Mem Experiment: mode={args.mode} seed={args.seed}")
+ print(f" model={args.model} memory_model={args.memory_model}")
+ print(f" num_tasks={args.num_tasks} retrieve_k={args.retrieve_k}")
+ print(f" model_settings={settings_kwargs or 'default'}")
+ print(f" benchmarks={benchmarks_to_run}")
+ print(f" output_dir={args.output_dir}")
+ print(f"{'=' * 70}\n")
+
+ task_order = get_unified_task_order(configs, args.num_tasks, args.seed, args.mode)
+ print(f"Total tasks: {len(task_order)}")
+
+ output_dir = Path(args.output_dir)
+ output_dir.mkdir(parents=True, exist_ok=True)
+ metrics_path = output_dir / "online_metrics.jsonl"
+
+ exp_config = {
+ "mode": args.mode,
+ "seed": args.seed,
+ "agent": "a_mem",
+ "model": args.model,
+ "memory_model": args.memory_model,
+ "retrieve_k": args.retrieve_k,
+ "evo_threshold": args.evo_threshold,
+ "embedding_model": args.embedding_model,
+ "num_tasks": args.num_tasks,
+ "benchmarks": benchmarks_to_run,
+ "task_order": [(s, t) for s, t in task_order],
+ }
+ with open(output_dir / "experiment_config.json", "w") as f:
+ json.dump(exp_config, f, indent=2)
+
+
+ MemoryStore.reset_all()
+ if args.mode == "interleaved":
+ _ckpt_ids = ["amem_interleaved_global"]
+ elif args.mode == "sequential":
+ _ckpt_ids = ["amem_sequential_global"]
+ else:
+ _ckpt_ids = [f"amem_isolated_{b}" for b in benchmarks_to_run]
+
+ _restored = False
+ _restored_session_count = 0
+ for sid in _ckpt_ids:
+ ckpt_path = output_dir / f"memory_{sid}.json"
+ if ckpt_path.exists():
+ store = MemoryStore.get_or_create(
+ shuffle_mode=args.mode,
+ benchmark_id=sid.replace("amem_isolated_", "") if args.mode == "isolated" else None,
+ embedding_model=args.embedding_model,
+ evo_threshold=args.evo_threshold,
+ )
+ store.load_checkpoint(str(ckpt_path))
+ _restored_session_count = max(_restored_session_count, store.session_count)
+ _restored = True
+ if _restored:
+ print(f" Restored memory from checkpoint (session_count={_restored_session_count})")
+
+ all_scores: list[float] = []
+ bm_scores: defaultdict[str, list[float]] = defaultdict(list)
+ session_index = 0
+
+ if metrics_path.exists():
+ kept_lines: list[str] = []
+ with open(metrics_path, "r") as f:
+ for line in f:
+ if _restored and session_index >= _restored_session_count:
+ break
+ rec = json.loads(line)
+ all_scores.append(rec["score"])
+ bm_scores[rec["benchmark_slug"]].append(rec["score"])
+ kept_lines.append(line)
+ session_index += 1
+ with open(metrics_path, "w") as f:
+ f.writelines(kept_lines)
+ if session_index > 0:
+ print(f" Restored {session_index} metrics records (cum_avg={sum(all_scores)/len(all_scores):.3f})")
+
+ if args.mode in ("isolated", "sequential"):
+ _completed_benchmarks: set[str] = set()
+ if _restored and session_index > 0:
+ _bm_counts: dict[str, int] = defaultdict(int)
+ with open(metrics_path, "r") as f:
+ for line in f:
+ rec = json.loads(line)
+ _bm_counts[rec["benchmark_slug"]] += 1
+ for bm_slug, task_ids in group_by_benchmark(task_order):
+ if _bm_counts.get(bm_slug, 0) >= len(task_ids):
+ _completed_benchmarks.add(bm_slug)
+ if _completed_benchmarks:
+ print(f" Skipping completed benchmarks: {sorted(_completed_benchmarks)}")
+
+ for bm_slug, task_ids in group_by_benchmark(task_order):
+ if bm_slug in _completed_benchmarks:
+ continue
+
+ print(f"\n{'=' * 60}")
+ print(f" {args.mode.upper()} — {bm_slug} ({len(task_ids)} tasks)")
+ print(f"{'=' * 60}\n")
+
+ bm_kwargs = configs[bm_slug]["bm_kwargs"]
+ agent_kwargs = configs[bm_slug].get("agent_kwargs", {})
+
+ benchmark = load_benchmark(bm_slug)(**bm_kwargs)
+ agent = load_agent("a_mem")(
+ model=args.model,
+ memory_model=args.memory_model,
+ shuffle_mode=args.mode,
+ benchmark_id=bm_slug,
+ retrieve_k=args.retrieve_k,
+ evo_threshold=args.evo_threshold,
+ embedding_model=args.embedding_model,
+ runner="direct",
+ model_settings=model_settings,
+ **agent_kwargs,
+ )
+
+ results = evaluate(
+ benchmark=benchmark,
+ agent=agent,
+ task_ids=task_ids,
+ max_workers=1,
+ output_dir=str(output_dir),
+ )
+
+ print(f" {bm_slug} score={results.benchmark_score}")
+
+ stores = MemoryStore.list_stores()
+ store_key = (f"amem_isolated_{bm_slug}" if args.mode == "isolated"
+ else "amem_sequential_global")
+ store = stores.get(store_key)
+ if store:
+ stats = store.get_stats()
+ print(f" memory: sessions={store.session_count}, "
+ f"memories={stats['total_memories']}, "
+ f"evolutions={stats['total_evolutions']}")
+ store.save_checkpoint(str(output_dir / f"memory_{store_key}.json"))
+
+ for i, sr in enumerate(results.session_results):
+ tid = task_ids[i] if i < len(task_ids) else sr.task_id or "?"
+ rec = record_online_metrics(
+ metrics_path, session_index, bm_slug, tid, sr,
+ args.mode, args.seed, args.model,
+ all_scores, bm_scores,
+ )
+ print(f" [{session_index}] {bm_slug}::{tid} "
+ f"score={rec['score']:.2f} cum={rec['cumulative_avg_score']:.3f} "
+ f"steps={rec['steps']}")
+ session_index += 1
+
+ elif args.mode == "interleaved":
+ for i, (bm_slug, task_id) in enumerate(task_order):
+ if _restored and i < _restored_session_count:
+ print(f" Skipping Interleaved [{i+1}/{len(task_order)}] {bm_slug}::{task_id} (cached)")
+ continue
+
+ print(f"\n--- Interleaved [{i+1}/{len(task_order)}] {bm_slug}::{task_id} ---")
+
+ bm_kwargs = configs[bm_slug]["bm_kwargs"]
+ agent_kwargs = configs[bm_slug].get("agent_kwargs", {})
+
+ benchmark = load_benchmark(bm_slug)(**bm_kwargs)
+ agent = load_agent("a_mem")(
+ model=args.model,
+ memory_model=args.memory_model,
+ shuffle_mode="interleaved",
+ benchmark_id=bm_slug,
+ retrieve_k=args.retrieve_k,
+ evo_threshold=args.evo_threshold,
+ embedding_model=args.embedding_model,
+ runner="direct",
+ model_settings=model_settings,
+ **agent_kwargs,
+ )
+
+ results = evaluate(
+ benchmark=benchmark,
+ agent=agent,
+ task_ids=[task_id],
+ max_workers=1,
+ output_dir=str(output_dir),
+ )
+
+ sr = results.session_results[0]
+ rec = record_online_metrics(
+ metrics_path, session_index, bm_slug, task_id, sr,
+ "interleaved", args.seed, args.model,
+ all_scores, bm_scores,
+ )
+ print(f" score={rec['score']:.2f} cum={rec['cumulative_avg_score']:.3f} "
+ f"steps={rec['steps']}")
+
+ stores = MemoryStore.list_stores()
+ store = stores.get("amem_interleaved_global")
+ if store:
+ stats = store.get_stats()
+ print(f" memory: sessions={store.session_count}, "
+ f"memories={stats['total_memories']}, "
+ f"evolutions={stats['total_evolutions']}")
+ store.save_checkpoint(str(output_dir / "memory_amem_interleaved_global.json"))
+ session_index += 1
+
+ print(f"\n{'=' * 70}")
+ print(f" Final Summary")
+ print(f"{'=' * 70}")
+ if all_scores:
+ print(f" Overall avg score: {sum(all_scores)/len(all_scores):.3f}")
+ for bm, scores in sorted(bm_scores.items()):
+ print(f" {bm:20s}: avg={sum(scores)/len(scores):.3f} n={len(scores)}")
+
+ stores = MemoryStore.list_stores()
+ for store_id, store in stores.items():
+ stats = store.get_stats()
+ print(f" memory[{store_id}]: sessions={store.session_count}, "
+ f"memories={stats['total_memories']}, "
+ f"evolutions={stats['total_evolutions']}, "
+ f"avg_links={stats['avg_links']:.2f}")
+
+ print(f"\n Metrics: {metrics_path}")
+ print(f" Output: {output_dir}")
+
+
+def main():
+ parser = argparse.ArgumentParser(description="Run A-Mem test-time learning experiment")
+ parser.add_argument("--mode", required=True, choices=["isolated", "sequential", "interleaved"])
+ parser.add_argument("--seed", type=int, required=True)
+ parser.add_argument("--num-tasks", type=int, default=50, help="Tasks per benchmark")
+ parser.add_argument("--model", default="openai/gpt-5.4")
+ parser.add_argument("--memory-model", default=None,
+ help="Model for memory evolution (defaults to --model)")
+ parser.add_argument("--benchmarks", default="browsecompplus,swebench,bfcl,tau2",
+ help="Comma-separated benchmark slugs")
+ parser.add_argument("--output-dir", required=True)
+ parser.add_argument("--max-tokens", type=int, default=None, help="Max output tokens")
+ parser.add_argument("--reasoning-effort", default=None, help="Reasoning effort (low/medium/high)")
+ # A-Mem specific hyperparameters
+ parser.add_argument("--retrieve-k", type=int, default=10,
+ help="Number of memories to retrieve per query")
+ parser.add_argument("--evo-threshold", type=int, default=100,
+ help="Consolidate embeddings every N evolutions")
+ parser.add_argument("--embedding-model", default="all-MiniLM-L6-v2",
+ help="SentenceTransformer model for memory retrieval")
+ args = parser.parse_args()
+
+ if args.memory_model is None:
+ args.memory_model = args.model
+
+ run_experiment(args)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/labs/AgentStream/exgentic/scripts/a_mem/run_experiment.sh b/labs/AgentStream/exgentic/scripts/a_mem/run_experiment.sh
new file mode 100644
index 00000000..e9036dd0
--- /dev/null
+++ b/labs/AgentStream/exgentic/scripts/a_mem/run_experiment.sh
@@ -0,0 +1,72 @@
+#!/bin/bash
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+set -e
+
+export OPENAI_API_BASE=""
+export OPENAI_API_KEY=""
+
+cd "$(dirname "$0")"
+
+# ============================================================
+# Configuration
+# ============================================================
+SEED=44
+NUM_TASKS=50
+MODEL="openai/gpt-5.4"
+MEMORY_MODEL="openai/gpt-5.4"
+MAX_TOKENS="default"
+REASONING_EFFORT="default"
+MODE="sequential" # isolated | sequential | interleaved
+
+# A-Mem specific hyperparameters
+RETRIEVE_K=10
+EVO_THRESHOLD=100
+EMBEDDING_MODEL="all-MiniLM-L6-v2"
+
+OUTPUT_BASE="./outputs"
+MODEL_SHORT=$(echo $MODEL | sed 's|openai/||; s|azure/||; s|/|_|g')
+RUN_TAG="amem_${MODE}_s${SEED}_${MODEL_SHORT}_${MAX_TOKENS}_${REASONING_EFFORT}"
+
+
+SETTINGS_ARGS=""
+[ "$MAX_TOKENS" != "default" ] && SETTINGS_ARGS="$SETTINGS_ARGS --max-tokens $MAX_TOKENS"
+[ "$REASONING_EFFORT" != "default" ] && SETTINGS_ARGS="$SETTINGS_ARGS --reasoning-effort $REASONING_EFFORT"
+echo "Mode: ${MODE}"
+echo "ModelSettings args: ${SETTINGS_ARGS:-default (no overrides)}"
+echo "A-Mem: retrieve_k=${RETRIEVE_K} evo_threshold=${EVO_THRESHOLD} embedding=${EMBEDDING_MODEL}"
+
+mkdir -p "$OUTPUT_BASE"
+
+ALL_BENCHMARKS="hle,bfcl,browsecompplus,appworld,swebench,tau2"
+if [ "$MODE" = "isolated" ]; then
+ for BENCH in swebench tau2 browsecompplus appworld hle bfcl; do
+ echo "=== Running isolated ${BENCH} ==="
+ uv run python run_experiment.py \
+ --mode isolated --seed $SEED --num-tasks $NUM_TASKS \
+ --model $MODEL --memory-model $MEMORY_MODEL \
+ --retrieve-k $RETRIEVE_K --evo-threshold $EVO_THRESHOLD \
+ --embedding-model $EMBEDDING_MODEL \
+ $SETTINGS_ARGS \
+ --benchmarks $BENCH \
+ --output-dir ${OUTPUT_BASE}/${RUN_TAG}_${BENCH} \
+ 2>&1 | tee ${OUTPUT_BASE}/${RUN_TAG}_${BENCH}.log
+ done
+else
+ echo "=== Running ${MODE} (all benchmarks) ==="
+ uv run python run_experiment.py \
+ --mode $MODE --seed $SEED --num-tasks $NUM_TASKS \
+ --model $MODEL --memory-model $MEMORY_MODEL \
+ --retrieve-k $RETRIEVE_K --evo-threshold $EVO_THRESHOLD \
+ --embedding-model $EMBEDDING_MODEL \
+ $SETTINGS_ARGS \
+ --benchmarks $ALL_BENCHMARKS \
+ --output-dir ${OUTPUT_BASE}/${RUN_TAG}_all \
+ 2>&1 | tee ${OUTPUT_BASE}/${RUN_TAG}_all.log
+fi
+
+echo ""
+echo "=== All benchmarks complete ==="
+echo "Run tag: ${RUN_TAG}"
+echo "Outputs in: ${OUTPUT_BASE}/${RUN_TAG}_*"
diff --git a/labs/AgentStream/exgentic/scripts/ace/run_experiment.py b/labs/AgentStream/exgentic/scripts/ace/run_experiment.py
new file mode 100644
index 00000000..b1ab9dba
--- /dev/null
+++ b/labs/AgentStream/exgentic/scripts/ace/run_experiment.py
@@ -0,0 +1,386 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from collections import defaultdict
+from datetime import datetime
+from pathlib import Path
+from typing import Any
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2] / "src"))
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "utils"))
+
+os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True"
+
+# --- ExGentic imports ---
+from exgentic.interfaces.lib.api import evaluate
+from exgentic.interfaces.registry import load_agent, load_benchmark
+from exgentic.agents.ace.playbook_store import PlaybookStore
+from exgentic.agents.ace.playbook_utils import get_playbook_stats
+from exgentic.core.types import ModelSettings
+
+from task_ordering import get_unified_task_order, group_by_benchmark
+
+
+BENCHMARK_REGISTRY: dict[str, dict[str, Any]] = {
+ "browsecompplus": {
+ "bm_kwargs": {
+ "searcher_type": "faiss",
+ "include_get_document": True,
+ "eval_model_id": "openai/gpt-5.4",
+ },
+ "agent_kwargs": {},
+ },
+ "swebench": {
+ "bm_kwargs": {
+ "subset": "princeton-nlp/SWE-bench_Verified",
+ },
+ "agent_kwargs": {},
+ },
+ "appworld": {
+ "bm_kwargs": {
+ "subset": "test_challenge",
+ },
+ "agent_kwargs": {
+ "enable_tool_shortlisting": True,
+ "max_selected_tools": 30,
+ },
+ },
+ "bfcl": {
+ "bm_kwargs": {
+ "subset": "multi_turn_base",
+ },
+ "agent_kwargs": {},
+ },
+ "tau2": {
+ "bm_kwargs": {
+ "subset": "telecom",
+ "user_simulator_model": "openai/gpt-5.4",
+ },
+ "agent_kwargs": {},
+ },
+ "hle": {
+ "bm_kwargs": {
+ "judge_model": "openai/gpt-5.4",
+ "runner": "direct",
+ },
+ "agent_kwargs": {},
+ },
+}
+
+def extract_token_counts(cost_reports: dict) -> tuple[int, int]:
+ """Extract total input/output tokens from cost_reports dict."""
+ total_in, total_out = 0, 0
+ for report in cost_reports.values():
+ if isinstance(report, dict):
+ total_in += report.get("input_tokens", 0)
+ total_out += report.get("output_tokens", 0)
+ elif hasattr(report, "input_tokens"):
+ total_in += report.input_tokens
+ total_out += report.output_tokens
+ return total_in, total_out
+
+
+def get_memory_tokens(mode: str, bm_slug: str) -> tuple[int, int]:
+ stores = PlaybookStore.list_stores()
+ if mode == "isolated":
+ store = stores.get(f"ace_isolated_{bm_slug}")
+ elif mode == "sequential":
+ store = stores.get("ace_sequential_global")
+ elif mode == "interleaved":
+ store = stores.get("ace_interleaved_global")
+ else:
+ return 0, 0
+ if store is None:
+ return 0, 0
+ stats = get_playbook_stats(store.playbook)
+ mem_tokens = len(store.playbook) // 4
+ return mem_tokens, stats["total_bullets"]
+
+
+
+def record_online_metrics(
+ metrics_path: Path,
+ session_index: int,
+ bm_slug: str,
+ task_id: str,
+ sr: Any,
+ mode: str,
+ seed: int,
+ model: str,
+ all_scores: list[float],
+ bm_scores: dict[str, list[float]],
+):
+ score = sr.score if sr.score is not None else (1.0 if sr.success else 0.0)
+ all_scores.append(score)
+ bm_scores[bm_slug].append(score)
+
+ input_tokens, output_tokens = extract_token_counts(sr.cost_reports)
+ memory_tokens, playbook_bullets = get_memory_tokens(mode, bm_slug)
+
+ record = {
+ "session_index": session_index,
+ "seed": seed,
+ "mode": mode,
+ "agent": "ace",
+ "model": model,
+ "benchmark_slug": bm_slug,
+ "task_id": task_id,
+ "score": score,
+ "cumulative_avg_score": sum(all_scores) / len(all_scores),
+ "benchmark_cumulative_avg_score": (
+ sum(bm_scores[bm_slug]) / len(bm_scores[bm_slug])
+ ),
+ "steps": sr.steps,
+ "action_count": sr.action_count,
+ "agent_cost": sr.agent_cost,
+ "input_tokens": input_tokens,
+ "output_tokens": output_tokens,
+ "memory_tokens": memory_tokens,
+ "playbook_bullets": playbook_bullets,
+ "execution_time": sr.execution_time,
+ "status": sr.status.value if hasattr(sr.status, "value") else str(sr.status),
+ "timestamp": datetime.now().isoformat(),
+ }
+
+ with open(metrics_path, "a", encoding="utf-8") as f:
+ f.write(json.dumps(record, ensure_ascii=False) + "\n")
+
+ return record
+
+def run_experiment(args):
+ benchmarks_to_run = [s.strip() for s in args.benchmarks.split(",")]
+ configs = {k: BENCHMARK_REGISTRY[k] for k in benchmarks_to_run}
+
+ settings_kwargs = {}
+ if args.max_tokens is not None:
+ settings_kwargs["max_tokens"] = args.max_tokens
+ if args.reasoning_effort is not None:
+ settings_kwargs["reasoning_effort"] = args.reasoning_effort
+ model_settings = ModelSettings(**settings_kwargs)
+
+ print(f"\n{'=' * 70}")
+ print(f" ACE Experiment: mode={args.mode} seed={args.seed}")
+ print(f" model={args.model} num_tasks={args.num_tasks}")
+ print(f" model_settings={settings_kwargs or 'default'}")
+ print(f" benchmarks={benchmarks_to_run}")
+ print(f" output_dir={args.output_dir}")
+ print(f"{'=' * 70}\n")
+
+ task_order = get_unified_task_order(configs, args.num_tasks, args.seed, args.mode)
+ print(f"Total tasks: {len(task_order)}")
+
+ output_dir = Path(args.output_dir)
+ output_dir.mkdir(parents=True, exist_ok=True)
+ metrics_path = output_dir / "online_metrics.jsonl"
+
+ exp_config = {
+ "mode": args.mode,
+ "seed": args.seed,
+ "agent": "ace",
+ "model": args.model,
+ "num_tasks": args.num_tasks,
+ "benchmarks": benchmarks_to_run,
+ "task_order": [(s, t) for s, t in task_order],
+ }
+ with open(output_dir / "experiment_config.json", "w") as f:
+ json.dump(exp_config, f, indent=2)
+
+ PlaybookStore.reset_all()
+ if args.mode == "interleaved":
+ _ckpt_ids = ["ace_interleaved_global"]
+ elif args.mode == "sequential":
+ _ckpt_ids = ["ace_sequential_global"]
+ else:
+ _ckpt_ids = [f"ace_isolated_{b}" for b in benchmarks_to_run]
+
+ _restored = False
+ _restored_session_count = 0
+ for sid in _ckpt_ids:
+ ckpt_path = output_dir / f"playbook_{sid}.json"
+ if ckpt_path.exists():
+ store = PlaybookStore.get_or_create(
+ shuffle_mode=args.mode,
+ benchmark_id=sid.replace("ace_isolated_", "") if args.mode == "isolated" else None,
+ )
+ store.load_checkpoint(str(ckpt_path))
+ _restored_session_count = max(_restored_session_count, store.session_count)
+ _restored = True
+ if _restored:
+ print(f" ♻️ Restored playbook from checkpoint (session_count={_restored_session_count})")
+
+ all_scores: list[float] = []
+ bm_scores: defaultdict[str, list[float]] = defaultdict(list)
+ session_index = 0
+
+ if metrics_path.exists():
+ kept_lines: list[str] = []
+ with open(metrics_path, "r") as f:
+ for line in f:
+ if _restored and session_index >= _restored_session_count:
+ break
+ rec = json.loads(line)
+ all_scores.append(rec["score"])
+ bm_scores[rec["benchmark_slug"]].append(rec["score"])
+ kept_lines.append(line)
+ session_index += 1
+ with open(metrics_path, "w") as f:
+ f.writelines(kept_lines)
+ if session_index > 0:
+ print(f" ♻️ Restored {session_index} metrics records (cum_avg={sum(all_scores)/len(all_scores):.3f})")
+
+ if args.mode in ("isolated", "sequential"):
+ _completed_benchmarks: set[str] = set()
+ if _restored and session_index > 0:
+ _bm_counts: dict[str, int] = defaultdict(int)
+ with open(metrics_path, "r") as f:
+ for line in f:
+ rec = json.loads(line)
+ _bm_counts[rec["benchmark_slug"]] += 1
+ for bm_slug, task_ids in group_by_benchmark(task_order):
+ if _bm_counts.get(bm_slug, 0) >= len(task_ids):
+ _completed_benchmarks.add(bm_slug)
+ if _completed_benchmarks:
+ print(f" ⏭️ Skipping completed benchmarks: {sorted(_completed_benchmarks)}")
+
+ for bm_slug, task_ids in group_by_benchmark(task_order):
+ if bm_slug in _completed_benchmarks:
+ continue
+
+ print(f"\n{'=' * 60}")
+ print(f" {args.mode.upper()} — {bm_slug} ({len(task_ids)} tasks)")
+ print(f"{'=' * 60}\n")
+
+ bm_kwargs = configs[bm_slug]["bm_kwargs"]
+ agent_kwargs = configs[bm_slug].get("agent_kwargs", {})
+
+ benchmark = load_benchmark(bm_slug)(**bm_kwargs)
+ agent = load_agent("ace")(
+ model=args.model,
+ curator_model=args.model,
+ shuffle_mode=args.mode,
+ benchmark_id=bm_slug,
+ runner="direct",
+ model_settings=model_settings,
+ **agent_kwargs,
+ )
+
+ results = evaluate(
+ benchmark=benchmark,
+ agent=agent,
+ task_ids=task_ids,
+ max_workers=1,
+ output_dir=str(output_dir),
+ )
+
+ print(f" {bm_slug} score={results.benchmark_score}")
+
+ stores = PlaybookStore.list_stores()
+ store_key = (f"ace_isolated_{bm_slug}" if args.mode == "isolated"
+ else "ace_sequential_global")
+ store = stores.get(store_key)
+ if store:
+ stats = get_playbook_stats(store.playbook)
+ print(f" playbook: sessions={store.session_count}, "
+ f"bullets={stats['total_bullets']}")
+ store.save_checkpoint(str(output_dir / f"playbook_{store_key}.json"))
+
+ for i, sr in enumerate(results.session_results):
+ tid = task_ids[i] if i < len(task_ids) else sr.task_id or "?"
+ rec = record_online_metrics(
+ metrics_path, session_index, bm_slug, tid, sr,
+ args.mode, args.seed, args.model,
+ all_scores, bm_scores,
+ )
+ print(f" [{session_index}] {bm_slug}::{tid} "
+ f"score={rec['score']:.2f} cum={rec['cumulative_avg_score']:.3f} "
+ f"steps={rec['steps']}")
+ session_index += 1
+
+ elif args.mode == "interleaved":
+ for i, (bm_slug, task_id) in enumerate(task_order):
+ if _restored and i < _restored_session_count:
+ print(f" ⏭️ Skipping Interleaved [{i+1}/{len(task_order)}] {bm_slug}::{task_id} (cached)")
+ continue
+
+ print(f"\n--- Interleaved [{i+1}/{len(task_order)}] {bm_slug}::{task_id} ---")
+
+ bm_kwargs = configs[bm_slug]["bm_kwargs"]
+ agent_kwargs = configs[bm_slug].get("agent_kwargs", {})
+
+ benchmark = load_benchmark(bm_slug)(**bm_kwargs)
+ agent = load_agent("ace")(
+ model=args.model,
+ curator_model=args.model,
+ shuffle_mode="interleaved",
+ benchmark_id=bm_slug,
+ runner="direct",
+ model_settings=model_settings,
+ **agent_kwargs,
+ )
+
+ results = evaluate(
+ benchmark=benchmark,
+ agent=agent,
+ task_ids=[task_id],
+ max_workers=1,
+ output_dir=str(output_dir),
+ )
+
+ sr = results.session_results[0]
+
+ rec = record_online_metrics(
+ metrics_path, session_index, bm_slug, task_id, sr,
+ "interleaved", args.seed, args.model,
+ all_scores, bm_scores,
+ )
+ print(f" score={rec['score']:.2f} cum={rec['cumulative_avg_score']:.3f} "
+ f"steps={rec['steps']}")
+ stores = PlaybookStore.list_stores()
+ store = stores.get("ace_interleaved_global")
+ if store:
+ stats = get_playbook_stats(store.playbook)
+ print(f" playbook: sessions={store.session_count}, "
+ f"bullets={stats['total_bullets']}")
+ store.save_checkpoint(str(output_dir / "playbook_ace_interleaved_global.json"))
+ session_index += 1
+
+ print(f"\n{'=' * 70}")
+ print(f" Final Summary")
+ print(f"{'=' * 70}")
+ print(f" Overall avg score: {sum(all_scores)/len(all_scores):.3f}")
+ for bm, scores in sorted(bm_scores.items()):
+ print(f" {bm:20s}: avg={sum(scores)/len(scores):.3f} n={len(scores)}")
+
+ stores = PlaybookStore.list_stores()
+ for store_id, store in stores.items():
+ stats = get_playbook_stats(store.playbook)
+ print(f" playbook[{store_id}]: sessions={store.session_count}, "
+ f"bullets={stats['total_bullets']}")
+
+ print(f"\n Metrics: {metrics_path}")
+ print(f" Output: {output_dir}")
+
+
+def main():
+ parser = argparse.ArgumentParser(description="Run ACE test-time learning experiment")
+ parser.add_argument("--mode", required=True, choices=["isolated", "sequential", "interleaved"])
+ parser.add_argument("--seed", type=int, required=True)
+ parser.add_argument("--num-tasks", type=int, default=50, help="Tasks per benchmark")
+ parser.add_argument("--model", default="openai/gpt-5.4")
+ parser.add_argument("--benchmarks", default="browsecompplus,swebench,bfcl,tau2",
+ help="Comma-separated benchmark slugs")
+ parser.add_argument("--output-dir", required=True)
+ parser.add_argument("--max-tokens", type=int, default=None, help="Max output tokens")
+ parser.add_argument("--reasoning-effort", default=None, help="Reasoning effort (low/medium/high)")
+ args = parser.parse_args()
+ run_experiment(args)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/labs/AgentStream/exgentic/scripts/ace/run_experiment.sh b/labs/AgentStream/exgentic/scripts/ace/run_experiment.sh
new file mode 100644
index 00000000..c1ad04cd
--- /dev/null
+++ b/labs/AgentStream/exgentic/scripts/ace/run_experiment.sh
@@ -0,0 +1,60 @@
+#!/bin/bash
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+set -e
+
+export OPENAI_API_BASE=""
+export OPENAI_API_KEY=""
+
+cd "$(dirname "$0")"
+
+# ============================================================
+# Configuration
+# ============================================================
+SEED=44
+NUM_TASKS=50
+MODEL="openai/gpt-5.4"
+MAX_TOKENS="default"
+REASONING_EFFORT="default"
+MODE="sequential" # isolated | sequential | interleaved
+
+
+OUTPUT_BASE="./outputs"
+MODEL_SHORT=$(echo $MODEL | sed 's|openai/||; s|azure/||; s|/|_|g')
+RUN_TAG="ace_${MODE}_s${SEED}_${MODEL_SHORT}_${MAX_TOKENS}_${REASONING_EFFORT}"
+
+SETTINGS_ARGS=""
+[ "$MAX_TOKENS" != "default" ] && SETTINGS_ARGS="$SETTINGS_ARGS --max-tokens $MAX_TOKENS"
+[ "$REASONING_EFFORT" != "default" ] && SETTINGS_ARGS="$SETTINGS_ARGS --reasoning-effort $REASONING_EFFORT"
+echo "Mode: ${MODE}"
+echo "ModelSettings args: ${SETTINGS_ARGS:-default (no overrides)}"
+
+mkdir -p "$OUTPUT_BASE"
+
+ALL_BENCHMARKS="hle,bfcl,browsecompplus,appworld,swebench,tau2"
+
+if [ "$MODE" = "isolated" ]; then
+ for BENCH in swebench tau2 browsecompplus appworld hle bfcl; do
+ echo "=== Running isolated ${BENCH} ==="
+ uv run python run_experiment.py \
+ --mode isolated --seed $SEED --num-tasks $NUM_TASKS \
+ --model $MODEL $SETTINGS_ARGS \
+ --benchmarks $BENCH \
+ --output-dir ${OUTPUT_BASE}/${RUN_TAG}_${BENCH} \
+ 2>&1 | tee ${OUTPUT_BASE}/${RUN_TAG}_${BENCH}.log
+ done
+else
+ echo "=== Running ${MODE} (all benchmarks) ==="
+ uv run python run_experiment.py \
+ --mode $MODE --seed $SEED --num-tasks $NUM_TASKS \
+ --model $MODEL $SETTINGS_ARGS \
+ --benchmarks $ALL_BENCHMARKS \
+ --output-dir ${OUTPUT_BASE}/${RUN_TAG}_all \
+ 2>&1 | tee ${OUTPUT_BASE}/${RUN_TAG}_all.log
+fi
+
+echo ""
+echo "=== All benchmarks complete ==="
+echo "Run tag: ${RUN_TAG}"
+echo "Outputs in: ${OUTPUT_BASE}/${RUN_TAG}_*"
diff --git a/labs/AgentStream/exgentic/scripts/autoskill/run_experiment.py b/labs/AgentStream/exgentic/scripts/autoskill/run_experiment.py
new file mode 100644
index 00000000..0c4f756b
--- /dev/null
+++ b/labs/AgentStream/exgentic/scripts/autoskill/run_experiment.py
@@ -0,0 +1,382 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from collections import defaultdict
+from datetime import datetime
+from pathlib import Path
+from typing import Any
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2] / "src"))
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "utils"))
+
+os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True"
+
+from exgentic.interfaces.lib.api import evaluate
+from exgentic.interfaces.registry import load_agent, load_benchmark
+from exgentic.agents.autoskill.skill_store import SkillStore
+from exgentic.core.types import ModelSettings
+
+from task_ordering import get_unified_task_order, group_by_benchmark
+
+BENCHMARK_REGISTRY: dict[str, dict[str, Any]] = {
+ "browsecompplus": {
+ "bm_kwargs": {
+ "searcher_type": "faiss",
+ "include_get_document": True,
+ "eval_model_id": "openai/gpt-5.4",
+ },
+ "agent_kwargs": {},
+ },
+ "swebench": {
+ "bm_kwargs": {
+ "subset": "princeton-nlp/SWE-bench_Verified",
+ },
+ "agent_kwargs": {},
+ },
+ "appworld": {
+ "bm_kwargs": {
+ "subset": "test_challenge",
+ },
+ "agent_kwargs": {
+ "enable_tool_shortlisting": True,
+ "max_selected_tools": 30,
+ },
+ },
+ "bfcl": {
+ "bm_kwargs": {
+ "subset": "multi_turn_base",
+ },
+ "agent_kwargs": {},
+ },
+ "tau2": {
+ "bm_kwargs": {
+ "subset": "telecom",
+ "user_simulator_model": "openai/gpt-5.4",
+ },
+ "agent_kwargs": {},
+ },
+ "hle": {
+ "bm_kwargs": {
+ "judge_model": "openai/gpt-5.4",
+ "runner": "direct",
+ },
+ "agent_kwargs": {},
+ },
+}
+
+def extract_token_counts(cost_reports: dict) -> tuple[int, int]:
+ """Extract total input/output tokens from cost_reports dict."""
+ total_in, total_out = 0, 0
+ for report in cost_reports.values():
+ if isinstance(report, dict):
+ total_in += report.get("input_tokens", 0)
+ total_out += report.get("output_tokens", 0)
+ elif hasattr(report, "input_tokens"):
+ total_in += report.input_tokens
+ total_out += report.output_tokens
+ return total_in, total_out
+
+
+def get_memory_tokens(mode: str, bm_slug: str) -> tuple[int, int]:
+ """Return (memory_tokens, skill_count) for current skill store state."""
+ stores = SkillStore.list_stores()
+ if mode == "isolated":
+ store = stores.get(f"autoskill_isolated_{bm_slug}")
+ elif mode == "sequential":
+ store = stores.get("autoskill_sequential_global")
+ elif mode == "interleaved":
+ store = stores.get("autoskill_interleaved_global")
+ else:
+ return 0, 0
+ if store is None:
+ return 0, 0
+ skills = store.list_skills()
+ total_chars = sum(len(s.to_search_text()) for s in skills)
+ mem_tokens = total_chars // 4
+ return mem_tokens, len(skills)
+
+def record_online_metrics(
+ metrics_path: Path,
+ session_index: int,
+ bm_slug: str,
+ task_id: str,
+ sr: Any,
+ mode: str,
+ seed: int,
+ model: str,
+ all_scores: list[float],
+ bm_scores: dict[str, list[float]],
+):
+ score = sr.score if sr.score is not None else (1.0 if sr.success else 0.0)
+ all_scores.append(score)
+ bm_scores[bm_slug].append(score)
+
+ input_tokens, output_tokens = extract_token_counts(sr.cost_reports)
+ memory_tokens, skill_count = get_memory_tokens(mode, bm_slug)
+
+ record = {
+ "session_index": session_index,
+ "seed": seed,
+ "mode": mode,
+ "agent": "autoskill",
+ "model": model,
+ "benchmark_slug": bm_slug,
+ "task_id": task_id,
+ "score": score,
+ "cumulative_avg_score": sum(all_scores) / len(all_scores),
+ "benchmark_cumulative_avg_score": (
+ sum(bm_scores[bm_slug]) / len(bm_scores[bm_slug])
+ ),
+ "steps": sr.steps,
+ "action_count": sr.action_count,
+ "agent_cost": sr.agent_cost,
+ "input_tokens": input_tokens,
+ "output_tokens": output_tokens,
+ "memory_tokens": memory_tokens,
+ "skill_count": skill_count,
+ "execution_time": sr.execution_time,
+ "status": sr.status.value if hasattr(sr.status, "value") else str(sr.status),
+ "timestamp": datetime.now().isoformat(),
+ }
+
+ with open(metrics_path, "a", encoding="utf-8") as f:
+ f.write(json.dumps(record, ensure_ascii=False) + "\n")
+
+ return record
+
+def run_experiment(args):
+ benchmarks_to_run = [s.strip() for s in args.benchmarks.split(",")]
+ configs = {k: BENCHMARK_REGISTRY[k] for k in benchmarks_to_run}
+
+ settings_kwargs = {}
+ if args.max_tokens is not None:
+ settings_kwargs["max_tokens"] = args.max_tokens
+ if args.reasoning_effort is not None:
+ settings_kwargs["reasoning_effort"] = args.reasoning_effort
+ model_settings = ModelSettings(**settings_kwargs)
+
+ print(f"\n{'=' * 70}")
+ print(f" AutoSkill Experiment: mode={args.mode} seed={args.seed}")
+ print(f" model={args.model} num_tasks={args.num_tasks}")
+ print(f" model_settings={settings_kwargs or 'default'}")
+ print(f" benchmarks={benchmarks_to_run}")
+ print(f" output_dir={args.output_dir}")
+ print(f"{'=' * 70}\n")
+
+ task_order = get_unified_task_order(configs, args.num_tasks, args.seed, args.mode)
+ print(f"Total tasks: {len(task_order)}")
+
+ output_dir = Path(args.output_dir)
+ output_dir.mkdir(parents=True, exist_ok=True)
+ metrics_path = output_dir / "online_metrics.jsonl"
+
+ exp_config = {
+ "mode": args.mode,
+ "seed": args.seed,
+ "agent": "autoskill",
+ "model": args.model,
+ "num_tasks": args.num_tasks,
+ "benchmarks": benchmarks_to_run,
+ "task_order": [(s, t) for s, t in task_order],
+ }
+ with open(output_dir / "experiment_config.json", "w") as f:
+ json.dump(exp_config, f, indent=2)
+
+ SkillStore.reset_all()
+ if args.mode == "interleaved":
+ _ckpt_ids = ["autoskill_interleaved_global"]
+ elif args.mode == "sequential":
+ _ckpt_ids = ["autoskill_sequential_global"]
+ else:
+ _ckpt_ids = [f"autoskill_isolated_{b}" for b in benchmarks_to_run]
+
+ _restored = False
+ _restored_session_count = 0
+ for sid in _ckpt_ids:
+ ckpt_path = output_dir / f"skillstore_{sid}.json"
+ if ckpt_path.exists():
+ store = SkillStore.get_or_create(
+ shuffle_mode=args.mode,
+ benchmark_id=sid.replace("autoskill_isolated_", "") if args.mode == "isolated" else None,
+ )
+ store.load_checkpoint(str(ckpt_path))
+ _restored_session_count = max(_restored_session_count, store.session_count)
+ _restored = True
+ if _restored:
+ print(f" ♻️ Restored skill store from checkpoint (session_count={_restored_session_count})")
+
+ all_scores: list[float] = []
+ bm_scores: defaultdict[str, list[float]] = defaultdict(list)
+ session_index = 0
+
+ if metrics_path.exists():
+ kept_lines: list[str] = []
+ with open(metrics_path, "r") as f:
+ for line in f:
+ if _restored and session_index >= _restored_session_count:
+ break
+ rec = json.loads(line)
+ all_scores.append(rec["score"])
+ bm_scores[rec["benchmark_slug"]].append(rec["score"])
+ kept_lines.append(line)
+ session_index += 1
+ with open(metrics_path, "w") as f:
+ f.writelines(kept_lines)
+ if session_index > 0:
+ print(f" ♻️ Restored {session_index} metrics records (cum_avg={sum(all_scores)/len(all_scores):.3f})")
+
+ if args.mode in ("isolated", "sequential"):
+ _completed_benchmarks: set[str] = set()
+ if _restored and session_index > 0:
+ _bm_counts: dict[str, int] = defaultdict(int)
+ with open(metrics_path, "r") as f:
+ for line in f:
+ rec = json.loads(line)
+ _bm_counts[rec["benchmark_slug"]] += 1
+ for bm_slug, task_ids in group_by_benchmark(task_order):
+ if _bm_counts.get(bm_slug, 0) >= len(task_ids):
+ _completed_benchmarks.add(bm_slug)
+ if _completed_benchmarks:
+ print(f" ⏭️ Skipping completed benchmarks: {sorted(_completed_benchmarks)}")
+
+ for bm_slug, task_ids in group_by_benchmark(task_order):
+ if bm_slug in _completed_benchmarks:
+ continue
+
+ print(f"\n{'=' * 60}")
+ print(f" {args.mode.upper()} — {bm_slug} ({len(task_ids)} tasks)")
+ print(f"{'=' * 60}\n")
+
+ bm_kwargs = configs[bm_slug]["bm_kwargs"]
+ agent_kwargs = configs[bm_slug].get("agent_kwargs", {})
+
+ benchmark = load_benchmark(bm_slug)(**bm_kwargs)
+ agent = load_agent("autoskill")(
+ model=args.model,
+ skill_model=args.model,
+ shuffle_mode=args.mode,
+ benchmark_id=bm_slug,
+ embedding_model="all-MiniLM-L6-v2",
+ runner="direct",
+ model_settings=model_settings,
+ **agent_kwargs,
+ )
+
+ results = evaluate(
+ benchmark=benchmark,
+ agent=agent,
+ task_ids=task_ids,
+ max_workers=1,
+ output_dir=str(output_dir),
+ )
+
+ print(f" {bm_slug} score={results.benchmark_score}")
+ stores = SkillStore.list_stores()
+ store_key = (f"autoskill_isolated_{bm_slug}" if args.mode == "isolated"
+ else "autoskill_sequential_global")
+ store = stores.get(store_key)
+ if store:
+ print(f" skillbank: sessions={store.session_count}, "
+ f"skills={store.skill_count}")
+ store.save_checkpoint(str(output_dir / f"skillstore_{store_key}.json"))
+
+ for i, sr in enumerate(results.session_results):
+ tid = task_ids[i] if i < len(task_ids) else sr.task_id or "?"
+ rec = record_online_metrics(
+ metrics_path, session_index, bm_slug, tid, sr,
+ args.mode, args.seed, args.model,
+ all_scores, bm_scores,
+ )
+ print(f" [{session_index}] {bm_slug}::{tid} "
+ f"score={rec['score']:.2f} cum={rec['cumulative_avg_score']:.3f} "
+ f"steps={rec['steps']}")
+ session_index += 1
+
+ elif args.mode == "interleaved":
+ for i, (bm_slug, task_id) in enumerate(task_order):
+ if _restored and i < _restored_session_count:
+ print(f" ⏭️ Skipping Interleaved [{i+1}/{len(task_order)}] {bm_slug}::{task_id} (cached)")
+ continue
+
+ print(f"\n--- Interleaved [{i+1}/{len(task_order)}] {bm_slug}::{task_id} ---")
+
+ bm_kwargs = configs[bm_slug]["bm_kwargs"]
+ agent_kwargs = configs[bm_slug].get("agent_kwargs", {})
+
+ benchmark = load_benchmark(bm_slug)(**bm_kwargs)
+ agent = load_agent("autoskill")(
+ model=args.model,
+ skill_model=args.model,
+ shuffle_mode="interleaved",
+ benchmark_id=bm_slug,
+ embedding_model="all-MiniLM-L6-v2",
+ runner="direct",
+ model_settings=model_settings,
+ **agent_kwargs,
+ )
+
+ results = evaluate(
+ benchmark=benchmark,
+ agent=agent,
+ task_ids=[task_id],
+ max_workers=1,
+ output_dir=str(output_dir),
+ )
+
+ sr = results.session_results[0]
+
+ rec = record_online_metrics(
+ metrics_path, session_index, bm_slug, task_id, sr,
+ "interleaved", args.seed, args.model,
+ all_scores, bm_scores,
+ )
+ print(f" score={rec['score']:.2f} cum={rec['cumulative_avg_score']:.3f} "
+ f"steps={rec['steps']}")
+
+ stores = SkillStore.list_stores()
+ store = stores.get("autoskill_interleaved_global")
+ if store:
+ print(f" skillbank: sessions={store.session_count}, "
+ f"skills={store.skill_count}")
+ store.save_checkpoint(str(output_dir / "skillstore_autoskill_interleaved_global.json"))
+ session_index += 1
+
+ print(f"\n{'=' * 70}")
+ print(f" Final Summary")
+ print(f"{'=' * 70}")
+ if all_scores:
+ print(f" Overall avg score: {sum(all_scores)/len(all_scores):.3f}")
+ for bm, scores in sorted(bm_scores.items()):
+ print(f" {bm:20s}: avg={sum(scores)/len(scores):.3f} n={len(scores)}")
+
+ stores = SkillStore.list_stores()
+ for store_id, store in stores.items():
+ print(f" skillbank[{store_id}]: sessions={store.session_count}, "
+ f"skills={store.skill_count}")
+
+ print(f"\n Metrics: {metrics_path}")
+ print(f" Output: {output_dir}")
+
+def main():
+ parser = argparse.ArgumentParser(description="Run AutoSkill test-time learning experiment")
+ parser.add_argument("--mode", required=True, choices=["isolated", "sequential", "interleaved"])
+ parser.add_argument("--seed", type=int, required=True)
+ parser.add_argument("--num-tasks", type=int, default=50, help="Tasks per benchmark")
+ parser.add_argument("--model", default="openai/gpt-5.4")
+ parser.add_argument("--benchmarks", default="browsecompplus,swebench,bfcl,tau2",
+ help="Comma-separated benchmark slugs")
+ parser.add_argument("--output-dir", required=True)
+ parser.add_argument("--max-tokens", type=int, default=None, help="Max output tokens")
+ parser.add_argument("--reasoning-effort", default=None, help="Reasoning effort (low/medium/high)")
+ args = parser.parse_args()
+ run_experiment(args)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/labs/AgentStream/exgentic/scripts/autoskill/run_experiment.sh b/labs/AgentStream/exgentic/scripts/autoskill/run_experiment.sh
new file mode 100644
index 00000000..86b59c1a
--- /dev/null
+++ b/labs/AgentStream/exgentic/scripts/autoskill/run_experiment.sh
@@ -0,0 +1,61 @@
+#!/bin/bash
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+set -e
+
+export OPENAI_API_BASE=""
+export OPENAI_API_KEY=""
+
+cd "$(dirname "$0")"
+
+# ============================================================
+# Configuration
+# ============================================================
+SEED=42
+NUM_TASKS=50
+MODEL="openai/gpt5.4"
+MAX_TOKENS="default"
+REASONING_EFFORT="default"
+MODE="sequential" # isolated | sequential | interleaved
+
+
+OUTPUT_BASE="./outputs"
+MODEL_SHORT=$(echo $MODEL | sed 's|openai/||; s|azure/||; s|anthropic/||; s|/|_|g')
+RUN_TAG="autoskill_${MODE}_s${SEED}_${MODEL_SHORT}_${MAX_TOKENS}_${REASONING_EFFORT}"
+
+SETTINGS_ARGS=""
+[ "$MAX_TOKENS" != "default" ] && SETTINGS_ARGS="$SETTINGS_ARGS --max-tokens $MAX_TOKENS"
+[ "$REASONING_EFFORT" != "default" ] && SETTINGS_ARGS="$SETTINGS_ARGS --reasoning-effort $REASONING_EFFORT"
+echo "Mode: ${MODE}"
+echo "Model: ${MODEL}"
+echo "ModelSettings args: ${SETTINGS_ARGS:-default (no overrides)}"
+
+mkdir -p "$OUTPUT_BASE"
+
+ALL_BENCHMARKS="hle,bfcl,browsecompplus,appworld,swebench,tau2"
+
+if [ "$MODE" = "isolated" ]; then
+ for BENCH in swebench tau2 browsecompplus appworld hle bfcl; do
+ echo "=== Running isolated ${BENCH} ==="
+ uv run python run_experiment.py \
+ --mode isolated --seed $SEED --num-tasks $NUM_TASKS \
+ --model $MODEL $SETTINGS_ARGS \
+ --benchmarks $BENCH \
+ --output-dir ${OUTPUT_BASE}/${RUN_TAG}_${BENCH} \
+ 2>&1 | tee ${OUTPUT_BASE}/${RUN_TAG}_${BENCH}.log
+ done
+else
+ echo "=== Running ${MODE} (all benchmarks) ==="
+ uv run python run_experiment.py \
+ --mode $MODE --seed $SEED --num-tasks $NUM_TASKS \
+ --model $MODEL $SETTINGS_ARGS \
+ --benchmarks $ALL_BENCHMARKS \
+ --output-dir ${OUTPUT_BASE}/${RUN_TAG}_all \
+ 2>&1 | tee ${OUTPUT_BASE}/${RUN_TAG}_all.log
+fi
+
+echo ""
+echo "=== All benchmarks complete ==="
+echo "Run tag: ${RUN_TAG}"
+echo "Outputs in: ${OUTPUT_BASE}/${RUN_TAG}_*"
diff --git a/labs/AgentStream/exgentic/scripts/harness/run_experiment.py b/labs/AgentStream/exgentic/scripts/harness/run_experiment.py
new file mode 100644
index 00000000..b211007c
--- /dev/null
+++ b/labs/AgentStream/exgentic/scripts/harness/run_experiment.py
@@ -0,0 +1,386 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from collections import defaultdict
+from datetime import datetime
+from pathlib import Path
+from typing import Any
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2] / "src"))
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "utils"))
+
+os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True"
+
+# --- ExGentic imports ---
+from exgentic.interfaces.lib.api import evaluate
+from exgentic.interfaces.registry import load_agent, load_benchmark
+from exgentic.agents.harness.harness_store import HarnessStore
+from exgentic.core.types import ModelSettings
+
+from task_ordering import get_unified_task_order, group_by_benchmark
+
+
+BENCHMARK_REGISTRY: dict[str, dict[str, Any]] = {
+ "browsecompplus": {
+ "bm_kwargs": {
+ "searcher_type": "faiss",
+ "include_get_document": True,
+ "eval_model_id": "openai/gpt-5.4",
+ },
+ "agent_kwargs": {},
+ },
+ "swebench": {
+ "bm_kwargs": {
+ "subset": "princeton-nlp/SWE-bench_Verified",
+ },
+ "agent_kwargs": {},
+ },
+ "appworld": {
+ "bm_kwargs": {
+ "subset": "test_challenge",
+ },
+ "agent_kwargs": {
+ "enable_tool_shortlisting": True,
+ "max_selected_tools": 30,
+ },
+ },
+ "bfcl": {
+ "bm_kwargs": {
+ "subset": "multi_turn_base",
+ },
+ "agent_kwargs": {},
+ },
+ "tau2": {
+ "bm_kwargs": {
+ "subset": "telecom",
+ "user_simulator_model": "openai/gpt-5.4",
+ },
+ "agent_kwargs": {},
+ },
+ "hle": {
+ "bm_kwargs": {
+ "judge_model": "openai/gpt-5.4",
+ "runner": "direct",
+ },
+ "agent_kwargs": {},
+ },
+}
+
+
+def extract_token_counts(cost_reports: dict) -> tuple[int, int]:
+ total_in, total_out = 0, 0
+ for report in cost_reports.values():
+ if isinstance(report, dict):
+ total_in += report.get("input_tokens", 0)
+ total_out += report.get("output_tokens", 0)
+ elif hasattr(report, "input_tokens"):
+ total_in += report.input_tokens
+ total_out += report.output_tokens
+ return total_in, total_out
+
+
+def get_memory_tokens(mode: str, bm_slug: str) -> tuple[int, int]:
+ stores = HarnessStore.list_stores()
+ if mode == "isolated":
+ store = stores.get(f"harness_isolated_{bm_slug}")
+ elif mode == "sequential":
+ store = stores.get("harness_sequential_global")
+ elif mode == "interleaved":
+ store = stores.get("harness_interleaved_global")
+ else:
+ return 0, 0
+ if store is None:
+ return 0, 0
+ total_chars = len(store.system_prompt) + len(store.memory)
+ for skill in store.list_skills():
+ total_chars += len(skill.description) + len(skill.body)
+ mem_tokens = total_chars // 4
+ return mem_tokens, store.skill_count
+
+
+def record_online_metrics(
+ metrics_path: Path,
+ session_index: int,
+ bm_slug: str,
+ task_id: str,
+ sr: Any,
+ mode: str,
+ seed: int,
+ model: str,
+ all_scores: list[float],
+ bm_scores: dict[str, list[float]],
+):
+ score = sr.score if sr.score is not None else (1.0 if sr.success else 0.0)
+ all_scores.append(score)
+ bm_scores[bm_slug].append(score)
+
+ input_tokens, output_tokens = extract_token_counts(sr.cost_reports)
+ memory_tokens, skill_count = get_memory_tokens(mode, bm_slug)
+
+ record = {
+ "session_index": session_index,
+ "seed": seed,
+ "mode": mode,
+ "agent": "harness",
+ "model": model,
+ "benchmark_slug": bm_slug,
+ "task_id": task_id,
+ "score": score,
+ "cumulative_avg_score": sum(all_scores) / len(all_scores),
+ "benchmark_cumulative_avg_score": (
+ sum(bm_scores[bm_slug]) / len(bm_scores[bm_slug])
+ ),
+ "steps": sr.steps,
+ "action_count": sr.action_count,
+ "agent_cost": sr.agent_cost,
+ "input_tokens": input_tokens,
+ "output_tokens": output_tokens,
+ "memory_tokens": memory_tokens,
+ "skill_count": skill_count,
+ "execution_time": sr.execution_time,
+ "status": sr.status.value if hasattr(sr.status, "value") else str(sr.status),
+ "timestamp": datetime.now().isoformat(),
+ }
+
+ with open(metrics_path, "a", encoding="utf-8") as f:
+ f.write(json.dumps(record, ensure_ascii=False) + "\n")
+
+ return record
+
+def run_experiment(args):
+ benchmarks_to_run = [s.strip() for s in args.benchmarks.split(",")]
+ configs = {k: BENCHMARK_REGISTRY[k] for k in benchmarks_to_run}
+
+ settings_kwargs = {}
+ if args.max_tokens is not None:
+ settings_kwargs["max_tokens"] = args.max_tokens
+ if args.reasoning_effort is not None:
+ settings_kwargs["reasoning_effort"] = args.reasoning_effort
+ model_settings = ModelSettings(**settings_kwargs)
+
+ print(f"\n{'=' * 70}")
+ print(f" Harness Experiment: mode={args.mode} seed={args.seed}")
+ print(f" model={args.model} num_tasks={args.num_tasks}")
+ print(f" model_settings={settings_kwargs or 'default'}")
+ print(f" benchmarks={benchmarks_to_run}")
+ print(f" output_dir={args.output_dir}")
+ print(f"{'=' * 70}\n")
+
+ task_order = get_unified_task_order(configs, args.num_tasks, args.seed, args.mode)
+ print(f"Total tasks: {len(task_order)}")
+
+ output_dir = Path(args.output_dir)
+ output_dir.mkdir(parents=True, exist_ok=True)
+ metrics_path = output_dir / "online_metrics.jsonl"
+
+ exp_config = {
+ "mode": args.mode,
+ "seed": args.seed,
+ "agent": "harness",
+ "model": args.model,
+ "num_tasks": args.num_tasks,
+ "benchmarks": benchmarks_to_run,
+ "task_order": [(s, t) for s, t in task_order],
+ }
+ with open(output_dir / "experiment_config.json", "w") as f:
+ json.dump(exp_config, f, indent=2)
+
+ HarnessStore.reset_all()
+ if args.mode == "interleaved":
+ _ckpt_ids = ["harness_interleaved_global"]
+ elif args.mode == "sequential":
+ _ckpt_ids = ["harness_sequential_global"]
+ else:
+ _ckpt_ids = [f"harness_isolated_{b}" for b in benchmarks_to_run]
+
+ _restored = False
+ _restored_session_count = 0
+ for sid in _ckpt_ids:
+ ckpt_path = output_dir / f"harness_{sid}.json"
+ if ckpt_path.exists():
+ store = HarnessStore.get_or_create(
+ shuffle_mode=args.mode,
+ benchmark_id=sid.replace("harness_isolated_", "") if args.mode == "isolated" else None,
+ )
+ store.load_checkpoint(str(ckpt_path))
+ _restored_session_count = max(_restored_session_count, store.session_count)
+ _restored = True
+ if _restored:
+ print(f" ♻️ Restored harness store from checkpoint (session_count={_restored_session_count})")
+
+ all_scores: list[float] = []
+ bm_scores: defaultdict[str, list[float]] = defaultdict(list)
+ session_index = 0
+
+ if metrics_path.exists():
+ kept_lines: list[str] = []
+ with open(metrics_path, "r") as f:
+ for line in f:
+ if _restored and session_index >= _restored_session_count:
+ break
+ rec = json.loads(line)
+ all_scores.append(rec["score"])
+ bm_scores[rec["benchmark_slug"]].append(rec["score"])
+ kept_lines.append(line)
+ session_index += 1
+ with open(metrics_path, "w") as f:
+ f.writelines(kept_lines)
+ if session_index > 0:
+ print(f" ♻️ Restored {session_index} metrics records (cum_avg={sum(all_scores)/len(all_scores):.3f})")
+
+ if args.mode in ("isolated", "sequential"):
+ _completed_benchmarks: set[str] = set()
+ if _restored and session_index > 0:
+ _bm_counts: dict[str, int] = defaultdict(int)
+ with open(metrics_path, "r") as f:
+ for line in f:
+ rec = json.loads(line)
+ _bm_counts[rec["benchmark_slug"]] += 1
+ for bm_slug, task_ids in group_by_benchmark(task_order):
+ if _bm_counts.get(bm_slug, 0) >= len(task_ids):
+ _completed_benchmarks.add(bm_slug)
+ if _completed_benchmarks:
+ print(f" ⏭️ Skipping completed benchmarks: {sorted(_completed_benchmarks)}")
+
+ for bm_slug, task_ids in group_by_benchmark(task_order):
+ if bm_slug in _completed_benchmarks:
+ continue
+
+ print(f"\n{'=' * 60}")
+ print(f" {args.mode.upper()} — {bm_slug} ({len(task_ids)} tasks)")
+ print(f"{'=' * 60}\n")
+
+ bm_kwargs = configs[bm_slug]["bm_kwargs"]
+ agent_kwargs = configs[bm_slug].get("agent_kwargs", {})
+
+ benchmark = load_benchmark(bm_slug)(**bm_kwargs)
+ agent = load_agent("harness")(
+ model=args.model,
+ evolver_model=args.model,
+ shuffle_mode=args.mode,
+ benchmark_id=bm_slug,
+ embedding_model="all-MiniLM-L6-v2",
+ runner="direct",
+ model_settings=model_settings,
+ **agent_kwargs,
+ )
+
+ results = evaluate(
+ benchmark=benchmark,
+ agent=agent,
+ task_ids=task_ids,
+ max_workers=1,
+ output_dir=str(output_dir),
+ )
+
+ print(f" {bm_slug} score={results.benchmark_score}")
+ stores = HarnessStore.list_stores()
+ store_key = (f"harness_isolated_{bm_slug}" if args.mode == "isolated"
+ else "harness_sequential_global")
+ store = stores.get(store_key)
+ if store:
+ print(f" harness: sessions={store.session_count}, "
+ f"skills={store.skill_count}")
+ store.save_checkpoint(str(output_dir / f"harness_{store_key}.json"))
+
+ for i, sr in enumerate(results.session_results):
+ tid = task_ids[i] if i < len(task_ids) else sr.task_id or "?"
+ rec = record_online_metrics(
+ metrics_path, session_index, bm_slug, tid, sr,
+ args.mode, args.seed, args.model,
+ all_scores, bm_scores,
+ )
+ print(f" [{session_index}] {bm_slug}::{tid} "
+ f"score={rec['score']:.2f} cum={rec['cumulative_avg_score']:.3f} "
+ f"steps={rec['steps']}")
+ session_index += 1
+
+ elif args.mode == "interleaved":
+ for i, (bm_slug, task_id) in enumerate(task_order):
+ if _restored and i < _restored_session_count:
+ print(f" ⏭️ Skipping Interleaved [{i+1}/{len(task_order)}] {bm_slug}::{task_id} (cached)")
+ continue
+
+ print(f"\n--- Interleaved [{i+1}/{len(task_order)}] {bm_slug}::{task_id} ---")
+
+ bm_kwargs = configs[bm_slug]["bm_kwargs"]
+ agent_kwargs = configs[bm_slug].get("agent_kwargs", {})
+
+ benchmark = load_benchmark(bm_slug)(**bm_kwargs)
+ agent = load_agent("harness")(
+ model=args.model,
+ evolver_model=args.model,
+ shuffle_mode="interleaved",
+ benchmark_id=bm_slug,
+ embedding_model="all-MiniLM-L6-v2",
+ runner="direct",
+ model_settings=model_settings,
+ **agent_kwargs,
+ )
+
+ results = evaluate(
+ benchmark=benchmark,
+ agent=agent,
+ task_ids=[task_id],
+ max_workers=1,
+ output_dir=str(output_dir),
+ )
+
+ sr = results.session_results[0]
+
+ rec = record_online_metrics(
+ metrics_path, session_index, bm_slug, task_id, sr,
+ "interleaved", args.seed, args.model,
+ all_scores, bm_scores,
+ )
+ print(f" score={rec['score']:.2f} cum={rec['cumulative_avg_score']:.3f} "
+ f"steps={rec['steps']}")
+
+ stores = HarnessStore.list_stores()
+ store = stores.get("harness_interleaved_global")
+ if store:
+ print(f" harness: sessions={store.session_count}, "
+ f"skills={store.skill_count}")
+ store.save_checkpoint(str(output_dir / "harness_harness_interleaved_global.json"))
+ session_index += 1
+
+ print(f"\n{'=' * 70}")
+ print(f" Final Summary")
+ print(f"{'=' * 70}")
+ if all_scores:
+ print(f" Overall avg score: {sum(all_scores)/len(all_scores):.3f}")
+ for bm, scores in sorted(bm_scores.items()):
+ print(f" {bm:20s}: avg={sum(scores)/len(scores):.3f} n={len(scores)}")
+
+ stores = HarnessStore.list_stores()
+ for store_id, store in stores.items():
+ print(f" harness[{store_id}]: sessions={store.session_count}, "
+ f"skills={store.skill_count}")
+
+ print(f"\n Metrics: {metrics_path}")
+ print(f" Output: {output_dir}")
+
+
+def main():
+ parser = argparse.ArgumentParser(description="Run Harness test-time learning experiment")
+ parser.add_argument("--mode", required=True, choices=["isolated", "sequential", "interleaved"])
+ parser.add_argument("--seed", type=int, required=True)
+ parser.add_argument("--num-tasks", type=int, default=50, help="Tasks per benchmark")
+ parser.add_argument("--model", default="openai/gpt-5.4")
+ parser.add_argument("--benchmarks", default="browsecompplus,swebench,bfcl,tau2",
+ help="Comma-separated benchmark slugs")
+ parser.add_argument("--output-dir", required=True)
+ parser.add_argument("--max-tokens", type=int, default=None, help="Max output tokens")
+ parser.add_argument("--reasoning-effort", default=None, help="Reasoning effort (low/medium/high)")
+ args = parser.parse_args()
+ run_experiment(args)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/labs/AgentStream/exgentic/scripts/harness/run_experiment.sh b/labs/AgentStream/exgentic/scripts/harness/run_experiment.sh
new file mode 100644
index 00000000..71ab5454
--- /dev/null
+++ b/labs/AgentStream/exgentic/scripts/harness/run_experiment.sh
@@ -0,0 +1,57 @@
+#!/bin/bash
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+set -e
+
+export OPENAI_API_BASE=""
+export OPENAI_API_KEY=""
+
+cd "$(dirname "$0")"
+
+SEED=44
+NUM_TASKS=50
+MODEL="openai/gpt-5.4"
+MAX_TOKENS="default"
+REASONING_EFFORT="default"
+MODE="sequential" # isolated | sequential | interleaved
+
+
+OUTPUT_BASE="./outputs"
+MODEL_SHORT=$(echo $MODEL | sed 's|openai/||; s|azure/||; s|/|_|g')
+RUN_TAG="harness_${MODE}_s${SEED}_${MODEL_SHORT}_${MAX_TOKENS}_${REASONING_EFFORT}"
+
+SETTINGS_ARGS=""
+[ "$MAX_TOKENS" != "default" ] && SETTINGS_ARGS="$SETTINGS_ARGS --max-tokens $MAX_TOKENS"
+[ "$REASONING_EFFORT" != "default" ] && SETTINGS_ARGS="$SETTINGS_ARGS --reasoning-effort $REASONING_EFFORT"
+echo "Mode: ${MODE}"
+echo "ModelSettings args: ${SETTINGS_ARGS:-default (no overrides)}"
+
+mkdir -p "$OUTPUT_BASE"
+
+ALL_BENCHMARKS="hle,bfcl,browsecompplus,appworld,swebench,tau2"
+
+if [ "$MODE" = "isolated" ]; then
+ for BENCH in swebench tau2 browsecompplus appworld hle bfcl; do
+ echo "=== Running isolated ${BENCH} ==="
+ uv run python run_experiment.py \
+ --mode isolated --seed $SEED --num-tasks $NUM_TASKS \
+ --model $MODEL $SETTINGS_ARGS \
+ --benchmarks $BENCH \
+ --output-dir ${OUTPUT_BASE}/${RUN_TAG}_${BENCH} \
+ 2>&1 | tee ${OUTPUT_BASE}/${RUN_TAG}_${BENCH}.log
+ done
+else
+ echo "=== Running ${MODE} (all benchmarks) ==="
+ uv run python run_experiment.py \
+ --mode $MODE --seed $SEED --num-tasks $NUM_TASKS \
+ --model $MODEL $SETTINGS_ARGS \
+ --benchmarks $ALL_BENCHMARKS \
+ --output-dir ${OUTPUT_BASE}/${RUN_TAG}_all \
+ 2>&1 | tee ${OUTPUT_BASE}/${RUN_TAG}_all.log
+fi
+
+echo ""
+echo "=== All benchmarks complete ==="
+echo "Run tag: ${RUN_TAG}"
+echo "Outputs in: ${OUTPUT_BASE}/${RUN_TAG}_*"
diff --git a/labs/AgentStream/exgentic/scripts/litellm/run_baseline.py b/labs/AgentStream/exgentic/scripts/litellm/run_baseline.py
new file mode 100644
index 00000000..4432741e
--- /dev/null
+++ b/labs/AgentStream/exgentic/scripts/litellm/run_baseline.py
@@ -0,0 +1,238 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+import time
+from collections import defaultdict
+from datetime import datetime
+from pathlib import Path
+from typing import Any
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2] / "src"))
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "ace"))
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "utils"))
+
+os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True"
+
+from exgentic.interfaces.lib.api import evaluate
+from exgentic.interfaces.registry import load_agent, load_benchmark
+from exgentic.core.types import ModelSettings
+
+from task_ordering import get_unified_task_order, group_by_benchmark
+
+
+BENCHMARK_REGISTRY: dict[str, dict[str, Any]] = {
+ "browsecompplus": {
+ "bm_kwargs": {
+ "searcher_type": "faiss",
+ "include_get_document": True,
+ "eval_model_id": "openai/gpt-5.4",
+ },
+ "agent_kwargs": {},
+ },
+ "swebench": {
+ "bm_kwargs": {
+ "subset": "princeton-nlp/SWE-bench_Verified",
+ },
+ "agent_kwargs": {},
+ },
+ "appworld": {
+ "bm_kwargs": {
+ "subset": "test_challenge",
+ },
+ "agent_kwargs": {
+ "enable_tool_shortlisting": True,
+ "max_selected_tools": 30,
+ },
+ },
+ "bfcl": {
+ "bm_kwargs": {
+ "subset": "multi_turn_base",
+ },
+ "agent_kwargs": {},
+ },
+ "tau2": {
+ "bm_kwargs": {
+ "subset": "telecom",
+ "user_simulator_model": "openai/gpt-5.4",
+ },
+ "agent_kwargs": {},
+ },
+ "hle": {
+ "bm_kwargs": {
+ "judge_model": "openai/gpt-5.4",
+ },
+ "agent_kwargs": {},
+ },
+}
+
+def extract_token_counts(cost_reports: dict) -> tuple[int, int]:
+ total_in, total_out = 0, 0
+ for report in cost_reports.values():
+ if isinstance(report, dict):
+ total_in += report.get("input_tokens", 0)
+ total_out += report.get("output_tokens", 0)
+ elif hasattr(report, "input_tokens"):
+ total_in += report.input_tokens
+ total_out += report.output_tokens
+ return total_in, total_out
+
+
+def record_online_metrics(
+ metrics_path: Path,
+ session_index: int,
+ bm_slug: str,
+ task_id: str,
+ sr: Any,
+ seed: int,
+ model: str,
+ all_scores: list[float],
+ bm_scores: dict[str, list[float]],
+):
+ score = sr.score if sr.score is not None else (1.0 if sr.success else 0.0)
+ all_scores.append(score)
+ bm_scores[bm_slug].append(score)
+
+ input_tokens, output_tokens = extract_token_counts(sr.cost_reports)
+
+ record = {
+ "session_index": session_index,
+ "seed": seed,
+ "mode": "baseline",
+ "agent": "tool_calling",
+ "model": model,
+ "benchmark_slug": bm_slug,
+ "task_id": task_id,
+ "score": score,
+ "cumulative_avg_score": sum(all_scores) / len(all_scores),
+ "benchmark_cumulative_avg_score": (
+ sum(bm_scores[bm_slug]) / len(bm_scores[bm_slug])
+ ),
+ "steps": sr.steps,
+ "action_count": sr.action_count,
+ "agent_cost": sr.agent_cost,
+ "input_tokens": input_tokens,
+ "output_tokens": output_tokens,
+ "memory_tokens": 0,
+ "playbook_bullets": 0,
+ "execution_time": sr.execution_time,
+ "status": sr.status.value if hasattr(sr.status, "value") else str(sr.status),
+ "timestamp": datetime.now().isoformat(),
+ }
+
+ with open(metrics_path, "a", encoding="utf-8") as f:
+ f.write(json.dumps(record, ensure_ascii=False) + "\n")
+
+ return record
+
+def run_baseline(args):
+ benchmarks_to_run = [s.strip() for s in args.benchmarks.split(",")]
+ configs = {k: BENCHMARK_REGISTRY[k] for k in benchmarks_to_run}
+
+ settings_kwargs = {}
+ if args.max_tokens is not None:
+ settings_kwargs["max_tokens"] = args.max_tokens
+ if args.reasoning_effort is not None:
+ settings_kwargs["reasoning_effort"] = args.reasoning_effort
+ model_settings = ModelSettings(**settings_kwargs)
+
+ print(f"\n{'=' * 70}")
+ print(f" Baseline: tool_calling agent (no learning)")
+ print(f" seed={args.seed} model={args.model} num_tasks={args.num_tasks}")
+ print(f" model_settings={settings_kwargs or 'default'}")
+ print(f" benchmarks={benchmarks_to_run}")
+ print(f" output_dir={args.output_dir}")
+ print(f"{'=' * 70}\n")
+
+ task_order = get_unified_task_order(configs, args.num_tasks, args.seed, "isolated")
+ print(f"Total tasks: {len(task_order)}")
+
+ output_dir = Path(args.output_dir)
+ output_dir.mkdir(parents=True, exist_ok=True)
+ metrics_path = output_dir / "online_metrics.jsonl"
+
+ exp_config = {
+ "mode": "baseline",
+ "seed": args.seed,
+ "agent": "tool_calling",
+ "model": args.model,
+ "num_tasks": args.num_tasks,
+ "benchmarks": benchmarks_to_run,
+ "task_order": [(s, t) for s, t in task_order],
+ }
+ with open(output_dir / "experiment_config.json", "w") as f:
+ json.dump(exp_config, f, indent=2)
+
+ all_scores: list[float] = []
+ bm_scores: defaultdict[str, list[float]] = defaultdict(list)
+ session_index = 0
+
+ for bm_slug, task_ids in group_by_benchmark(task_order):
+ print(f"\n{'=' * 60}")
+ print(f" BASELINE — {bm_slug} ({len(task_ids)} tasks)")
+ print(f"{'=' * 60}\n")
+
+ bm_kwargs = configs[bm_slug]["bm_kwargs"]
+ agent_kwargs = configs[bm_slug].get("agent_kwargs", {})
+
+ benchmark = load_benchmark(bm_slug)(**bm_kwargs)
+ agent = load_agent("tool_calling")(
+ model=args.model,
+ runner="direct",
+ model_settings=model_settings,
+ allow_truncated_messages=True,
+ **agent_kwargs,
+ )
+
+ results = evaluate(
+ benchmark=benchmark,
+ agent=agent,
+ task_ids=task_ids,
+ max_workers=5,
+ output_dir=str(output_dir),
+ )
+
+ print(f" {bm_slug} score={results.benchmark_score}")
+
+ for i, sr in enumerate(results.session_results):
+ tid = task_ids[i] if i < len(task_ids) else sr.task_id or "?"
+ rec = record_online_metrics(
+ metrics_path, session_index, bm_slug, tid, sr,
+ args.seed, args.model, all_scores, bm_scores,
+ )
+ print(f" [{session_index}] {bm_slug}::{tid} "
+ f"score={rec['score']:.2f} cum={rec['cumulative_avg_score']:.3f} "
+ f"steps={rec['steps']}")
+ session_index += 1
+
+ print(f"\n{'=' * 70}")
+ print(f" Baseline Summary")
+ print(f"{'=' * 70}")
+ print(f" Overall avg score: {sum(all_scores)/len(all_scores):.3f}")
+ for bm, scores in sorted(bm_scores.items()):
+ print(f" {bm:20s}: avg={sum(scores)/len(scores):.3f} n={len(scores)}")
+ print(f"\n Metrics: {metrics_path}")
+ print(f" Output: {output_dir}")
+
+
+def main():
+ parser = argparse.ArgumentParser(description="Run baseline (tool_calling agent, no learning)")
+ parser.add_argument("--seed", type=int, required=True)
+ parser.add_argument("--num-tasks", type=int, default=50, help="Tasks per benchmark")
+ parser.add_argument("--model", default="openai/gpt-5.4")
+ parser.add_argument("--benchmarks", default="browsecompplus,swebench,bfcl,tau2",
+ help="Comma-separated benchmark slugs")
+ parser.add_argument("--output-dir", required=True)
+ parser.add_argument("--max-tokens", type=int, default=None, help="Max output tokens")
+ parser.add_argument("--reasoning-effort", default=None, help="Reasoning effort (low/medium/high)")
+ args = parser.parse_args()
+ run_baseline(args)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/labs/AgentStream/exgentic/scripts/litellm/run_baseline.sh b/labs/AgentStream/exgentic/scripts/litellm/run_baseline.sh
new file mode 100644
index 00000000..382fa7e2
--- /dev/null
+++ b/labs/AgentStream/exgentic/scripts/litellm/run_baseline.sh
@@ -0,0 +1,84 @@
+#!/bin/bash
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+set -e
+
+export OPENAI_API_BASE=""
+export OPENAI_API_KEY=""
+
+cd "$(dirname "$0")"
+
+SEED=42
+NUM_TASKS=50
+MODEL="openai/gpt-5.4"
+MAX_TOKENS="default"
+REASONING_EFFORT="default"
+
+OUTPUT_BASE="./outputs"
+MODEL_SHORT=$(echo $MODEL | sed 's|openai/||; s|azure/||; s|/|_|g')
+RUN_TAG="baseline_s${SEED}_${MODEL_SHORT}_${MAX_TOKENS}_${REASONING_EFFORT}"
+
+SETTINGS_ARGS=""
+[ "$MAX_TOKENS" != "default" ] && SETTINGS_ARGS="$SETTINGS_ARGS --max-tokens $MAX_TOKENS"
+[ "$REASONING_EFFORT" != "default" ] && SETTINGS_ARGS="$SETTINGS_ARGS --reasoning-effort $REASONING_EFFORT"
+echo "ModelSettings args: ${SETTINGS_ARGS:-default (no overrides)}"
+
+# HLE
+echo "=== Running HLE ==="
+uv run python run_baseline.py \
+ --seed $SEED --num-tasks $NUM_TASKS \
+ --model $MODEL $SETTINGS_ARGS \
+ --benchmarks hle \
+ --output-dir ${OUTPUT_BASE}/${RUN_TAG}_hle \
+ 2>&1 | tee ${OUTPUT_BASE}/${RUN_TAG}_hle.log
+
+# BFCL
+echo "=== Running BFCL ==="
+uv run python run_baseline.py \
+ --seed $SEED --num-tasks $NUM_TASKS \
+ --model $MODEL $SETTINGS_ARGS \
+ --benchmarks bfcl \
+ --output-dir ${OUTPUT_BASE}/${RUN_TAG}_bfcl \
+ 2>&1 | tee ${OUTPUT_BASE}/${RUN_TAG}_bfcl.log
+
+# Tau2
+echo "=== Running Tau2 ==="
+uv run python run_baseline.py \
+ --seed $SEED --num-tasks $NUM_TASKS \
+ --model $MODEL $SETTINGS_ARGS \
+ --benchmarks tau2 \
+ --output-dir ${OUTPUT_BASE}/${RUN_TAG}_tau2 \
+ 2>&1 | tee ${OUTPUT_BASE}/${RUN_TAG}_tau2.log
+
+# BrowseCompPlus
+echo "=== Running BrowseCompPlus ==="
+uv run python run_baseline.py \
+ --seed $SEED --num-tasks $NUM_TASKS \
+ --model $MODEL $SETTINGS_ARGS \
+ --benchmarks browsecompplus \
+ --output-dir ${OUTPUT_BASE}/${RUN_TAG}_browsecompplus \
+ 2>&1 | tee ${OUTPUT_BASE}/${RUN_TAG}_browsecompplus.log
+
+# AppWorld
+echo "=== Running AppWorld ==="
+uv run python run_baseline.py \
+ --seed $SEED --num-tasks $NUM_TASKS \
+ --model $MODEL $SETTINGS_ARGS \
+ --benchmarks appworld \
+ --output-dir ${OUTPUT_BASE}/${RUN_TAG}_appworld \
+ 2>&1 | tee ${OUTPUT_BASE}/${RUN_TAG}_appworld.log
+
+# SWE-bench
+echo "=== Running SWE-bench ==="
+uv run python run_baseline.py \
+ --seed $SEED --num-tasks $NUM_TASKS \
+ --model $MODEL $SETTINGS_ARGS \
+ --benchmarks swebench \
+ --output-dir ${OUTPUT_BASE}/${RUN_TAG}_swebench \
+ 2>&1 | tee ${OUTPUT_BASE}/${RUN_TAG}_swebench.log
+
+echo ""
+echo "=== All benchmarks complete ==="
+echo "Run tag: ${RUN_TAG}"
+echo "Outputs in: ${OUTPUT_BASE}/${RUN_TAG}_*"
diff --git a/labs/AgentStream/exgentic/scripts/reasoning_bank/run_experiment.py b/labs/AgentStream/exgentic/scripts/reasoning_bank/run_experiment.py
new file mode 100644
index 00000000..6c49a725
--- /dev/null
+++ b/labs/AgentStream/exgentic/scripts/reasoning_bank/run_experiment.py
@@ -0,0 +1,403 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from collections import defaultdict
+from datetime import datetime
+from pathlib import Path
+from typing import Any
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[2] / "src"))
+sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "utils"))
+
+os.environ["LITELLM_LOCAL_MODEL_COST_MAP"] = "True"
+
+# --- ExGentic imports ---
+from exgentic.interfaces.lib.api import evaluate
+from exgentic.interfaces.registry import load_agent, load_benchmark
+from exgentic.agents.reasoning_bank.rb_store import ReasoningBankStore
+from exgentic.core.types import ModelSettings
+
+from task_ordering import get_unified_task_order, group_by_benchmark
+
+BENCHMARK_REGISTRY: dict[str, dict[str, Any]] = {
+ "browsecompplus": {
+ "bm_kwargs": {
+ "searcher_type": "faiss",
+ "include_get_document": True,
+ "eval_model_id": "openai/gpt-5.4",
+ },
+ "agent_kwargs": {},
+ },
+ "swebench": {
+ "bm_kwargs": {
+ "subset": "princeton-nlp/SWE-bench_Verified",
+ },
+ "agent_kwargs": {},
+ },
+ "appworld": {
+ "bm_kwargs": {
+ "subset": "test_challenge",
+ },
+ "agent_kwargs": {
+ "enable_tool_shortlisting": True,
+ "max_selected_tools": 30,
+ },
+ },
+ "bfcl": {
+ "bm_kwargs": {
+ "subset": "multi_turn_base",
+ },
+ "agent_kwargs": {},
+ },
+ "tau2": {
+ "bm_kwargs": {
+ "subset": "telecom",
+ "user_simulator_model": "openai/gpt-5.4",
+ },
+ "agent_kwargs": {},
+ },
+ "hle": {
+ "bm_kwargs": {
+ "judge_model": "openai/gpt-5.4",
+ "runner": "direct",
+ },
+ "agent_kwargs": {},
+ },
+}
+
+def extract_token_counts(cost_reports: dict) -> tuple[int, int]:
+ total_in, total_out = 0, 0
+ for report in cost_reports.values():
+ if isinstance(report, dict):
+ total_in += report.get("input_tokens", 0)
+ total_out += report.get("output_tokens", 0)
+ elif hasattr(report, "input_tokens"):
+ total_in += report.input_tokens
+ total_out += report.output_tokens
+ return total_in, total_out
+
+
+def get_memory_stats(mode: str, bm_slug: str) -> tuple[int, int]:
+ stores = ReasoningBankStore.list_stores()
+ if mode == "isolated":
+ store = stores.get(f"rb_isolated_{bm_slug}")
+ elif mode == "sequential":
+ store = stores.get("rb_sequential_global")
+ elif mode == "interleaved":
+ store = stores.get("rb_interleaved_global")
+ else:
+ return 0, 0
+ if store is None:
+ return 0, 0
+ entries = store.get_entries()
+ total_chars = sum(len("\n\n".join(e.memory_items)) for e in entries)
+ mem_tokens = total_chars // 4
+ return len(entries), mem_tokens
+
+
+
+def record_online_metrics(
+ metrics_path: Path,
+ session_index: int,
+ bm_slug: str,
+ task_id: str,
+ sr: Any,
+ mode: str,
+ seed: int,
+ model: str,
+ all_scores: list[float],
+ bm_scores: dict[str, list[float]],
+):
+ score = sr.score if sr.score is not None else (1.0 if sr.success else 0.0)
+ all_scores.append(score)
+ bm_scores[bm_slug].append(score)
+
+ input_tokens, output_tokens = extract_token_counts(sr.cost_reports)
+ memory_entries, memory_tokens = get_memory_stats(mode, bm_slug)
+
+ record = {
+ "session_index": session_index,
+ "seed": seed,
+ "mode": mode,
+ "agent": "reasoning_bank",
+ "model": model,
+ "benchmark_slug": bm_slug,
+ "task_id": task_id,
+ "score": score,
+ "cumulative_avg_score": sum(all_scores) / len(all_scores),
+ "benchmark_cumulative_avg_score": (
+ sum(bm_scores[bm_slug]) / len(bm_scores[bm_slug])
+ ),
+ "steps": sr.steps,
+ "action_count": sr.action_count,
+ "agent_cost": sr.agent_cost,
+ "input_tokens": input_tokens,
+ "output_tokens": output_tokens,
+ "memory_entries": memory_entries,
+ "memory_tokens": memory_tokens,
+ "execution_time": sr.execution_time,
+ "status": sr.status.value if hasattr(sr.status, "value") else str(sr.status),
+ "timestamp": datetime.now().isoformat(),
+ }
+
+ with open(metrics_path, "a", encoding="utf-8") as f:
+ f.write(json.dumps(record, ensure_ascii=False) + "\n")
+
+ return record
+
+
+def run_experiment(args):
+ benchmarks_to_run = [s.strip() for s in args.benchmarks.split(",")]
+ configs = {k: BENCHMARK_REGISTRY[k] for k in benchmarks_to_run}
+
+ settings_kwargs = {}
+ if args.max_tokens is not None:
+ settings_kwargs["max_tokens"] = args.max_tokens
+ if args.reasoning_effort is not None:
+ settings_kwargs["reasoning_effort"] = args.reasoning_effort
+ model_settings = ModelSettings(**settings_kwargs)
+
+ print(f"\n{'=' * 70}")
+ print(f" ReasoningBank Experiment: mode={args.mode} seed={args.seed}")
+ print(f" model={args.model} num_tasks={args.num_tasks}")
+ print(f" embedding_model={args.embedding_model}")
+ print(f" top_k_memories={args.top_k_memories} max_memory_items={args.max_memory_items}")
+ print(f" model_settings={settings_kwargs or 'default'}")
+ print(f" benchmarks={benchmarks_to_run}")
+ print(f" output_dir={args.output_dir}")
+ print(f"{'=' * 70}\n")
+
+ task_order = get_unified_task_order(configs, args.num_tasks, args.seed, args.mode)
+ print(f"Total tasks: {len(task_order)}")
+
+ output_dir = Path(args.output_dir)
+ output_dir.mkdir(parents=True, exist_ok=True)
+ metrics_path = output_dir / "online_metrics.jsonl"
+
+ exp_config = {
+ "mode": args.mode,
+ "seed": args.seed,
+ "agent": "reasoning_bank",
+ "model": args.model,
+ "embedding_model": args.embedding_model,
+ "top_k_memories": args.top_k_memories,
+ "max_memory_items": args.max_memory_items,
+ "num_tasks": args.num_tasks,
+ "benchmarks": benchmarks_to_run,
+ "task_order": [(s, t) for s, t in task_order],
+ }
+ with open(output_dir / "experiment_config.json", "w") as f:
+ json.dump(exp_config, f, indent=2)
+
+ ReasoningBankStore.reset_all()
+ if args.mode == "interleaved":
+ _ckpt_ids = ["rb_interleaved_global"]
+ elif args.mode == "sequential":
+ _ckpt_ids = ["rb_sequential_global"]
+ else:
+ _ckpt_ids = [f"rb_isolated_{b}" for b in benchmarks_to_run]
+
+ _restored = False
+ _restored_session_count = 0
+ for sid in _ckpt_ids:
+ ckpt_path = output_dir / f"store_{sid}.json"
+ if ckpt_path.exists():
+ store = ReasoningBankStore.get_or_create(
+ shuffle_mode=args.mode,
+ benchmark_id=sid.replace("rb_isolated_", "") if args.mode == "isolated" else None,
+ )
+ store.load_checkpoint(str(ckpt_path))
+ _restored_session_count = max(_restored_session_count, store.session_count)
+ _restored = True
+ if _restored:
+ print(f" Restored store from checkpoint (session_count={_restored_session_count})")
+
+ all_scores: list[float] = []
+ bm_scores: defaultdict[str, list[float]] = defaultdict(list)
+ session_index = 0
+
+ if metrics_path.exists():
+ kept_lines: list[str] = []
+ with open(metrics_path, "r") as f:
+ for line in f:
+ if _restored and session_index >= _restored_session_count:
+ break
+ rec = json.loads(line)
+ all_scores.append(rec["score"])
+ bm_scores[rec["benchmark_slug"]].append(rec["score"])
+ kept_lines.append(line)
+ session_index += 1
+ with open(metrics_path, "w") as f:
+ f.writelines(kept_lines)
+ if session_index > 0:
+ print(f" Restored {session_index} metrics records (cum_avg={sum(all_scores)/len(all_scores):.3f})")
+
+ if args.mode in ("isolated", "sequential"):
+ _completed_benchmarks: set[str] = set()
+ if _restored and session_index > 0:
+ _bm_counts: dict[str, int] = defaultdict(int)
+ with open(metrics_path, "r") as f:
+ for line in f:
+ rec = json.loads(line)
+ _bm_counts[rec["benchmark_slug"]] += 1
+ for bm_slug, task_ids in group_by_benchmark(task_order):
+ if _bm_counts.get(bm_slug, 0) >= len(task_ids):
+ _completed_benchmarks.add(bm_slug)
+ if _completed_benchmarks:
+ print(f" Skipping completed benchmarks: {sorted(_completed_benchmarks)}")
+
+ for bm_slug, task_ids in group_by_benchmark(task_order):
+ if bm_slug in _completed_benchmarks:
+ continue
+
+ print(f"\n{'=' * 60}")
+ print(f" {args.mode.upper()} -- {bm_slug} ({len(task_ids)} tasks)")
+ print(f"{'=' * 60}\n")
+
+ bm_kwargs = configs[bm_slug]["bm_kwargs"]
+ agent_kwargs = configs[bm_slug].get("agent_kwargs", {})
+
+ benchmark = load_benchmark(bm_slug)(**bm_kwargs)
+ agent = load_agent("reasoning_bank")(
+ model=args.model,
+ memory_model=args.model,
+ eval_model=args.model,
+ embedding_model=args.embedding_model,
+ top_k_memories=args.top_k_memories,
+ max_memory_items=args.max_memory_items,
+ shuffle_mode=args.mode,
+ benchmark_id=bm_slug,
+ runner="direct",
+ model_settings=model_settings,
+ **agent_kwargs,
+ )
+
+ results = evaluate(
+ benchmark=benchmark,
+ agent=agent,
+ task_ids=task_ids,
+ max_workers=1,
+ output_dir=str(output_dir),
+ )
+
+ print(f" {bm_slug} score={results.benchmark_score}")
+
+ stores = ReasoningBankStore.list_stores()
+ store_key = (f"rb_isolated_{bm_slug}" if args.mode == "isolated"
+ else "rb_sequential_global")
+ store = stores.get(store_key)
+ if store:
+ print(f" store: sessions={store.session_count}, "
+ f"entries={store.entry_count}")
+ store.save_checkpoint(str(output_dir / f"store_{store_key}.json"))
+
+ for i, sr in enumerate(results.session_results):
+ tid = task_ids[i] if i < len(task_ids) else sr.task_id or "?"
+ rec = record_online_metrics(
+ metrics_path, session_index, bm_slug, tid, sr,
+ args.mode, args.seed, args.model,
+ all_scores, bm_scores,
+ )
+ print(f" [{session_index}] {bm_slug}::{tid} "
+ f"score={rec['score']:.2f} cum={rec['cumulative_avg_score']:.3f} "
+ f"steps={rec['steps']}")
+ session_index += 1
+
+ elif args.mode == "interleaved":
+ for i, (bm_slug, task_id) in enumerate(task_order):
+ if _restored and i < _restored_session_count:
+ print(f" Skipping Interleaved [{i+1}/{len(task_order)}] {bm_slug}::{task_id} (cached)")
+ continue
+
+ print(f"\n--- Interleaved [{i+1}/{len(task_order)}] {bm_slug}::{task_id} ---")
+
+ bm_kwargs = configs[bm_slug]["bm_kwargs"]
+ agent_kwargs = configs[bm_slug].get("agent_kwargs", {})
+
+ benchmark = load_benchmark(bm_slug)(**bm_kwargs)
+ agent = load_agent("reasoning_bank")(
+ model=args.model,
+ memory_model=args.model,
+ eval_model=args.model,
+ embedding_model=args.embedding_model,
+ top_k_memories=args.top_k_memories,
+ max_memory_items=args.max_memory_items,
+ shuffle_mode="interleaved",
+ benchmark_id=bm_slug,
+ runner="direct",
+ model_settings=model_settings,
+ **agent_kwargs,
+ )
+
+ results = evaluate(
+ benchmark=benchmark,
+ agent=agent,
+ task_ids=[task_id],
+ max_workers=1,
+ output_dir=str(output_dir),
+ )
+
+ sr = results.session_results[0]
+
+ rec = record_online_metrics(
+ metrics_path, session_index, bm_slug, task_id, sr,
+ "interleaved", args.seed, args.model,
+ all_scores, bm_scores,
+ )
+ print(f" score={rec['score']:.2f} cum={rec['cumulative_avg_score']:.3f} "
+ f"steps={rec['steps']}")
+
+ stores = ReasoningBankStore.list_stores()
+ store = stores.get("rb_interleaved_global")
+ if store:
+ print(f" store: sessions={store.session_count}, "
+ f"entries={store.entry_count}")
+ store.save_checkpoint(str(output_dir / "store_rb_interleaved_global.json"))
+ session_index += 1
+
+ print(f"\n{'=' * 70}")
+ print(f" Final Summary")
+ print(f"{'=' * 70}")
+ if all_scores:
+ print(f" Overall avg score: {sum(all_scores)/len(all_scores):.3f}")
+ for bm, scores in sorted(bm_scores.items()):
+ print(f" {bm:20s}: avg={sum(scores)/len(scores):.3f} n={len(scores)}")
+
+ stores = ReasoningBankStore.list_stores()
+ for store_id, store in stores.items():
+ print(f" store[{store_id}]: sessions={store.session_count}, "
+ f"entries={store.entry_count}")
+
+ print(f"\n Metrics: {metrics_path}")
+ print(f" Output: {output_dir}")
+
+def main():
+ parser = argparse.ArgumentParser(description="Run ReasoningBank test-time learning experiment")
+ parser.add_argument("--mode", required=True, choices=["isolated", "sequential", "interleaved"])
+ parser.add_argument("--seed", type=int, required=True)
+ parser.add_argument("--num-tasks", type=int, default=50, help="Tasks per benchmark")
+ parser.add_argument("--model", default="openai/gpt-5.4")
+ parser.add_argument("--embedding-model", default="all-MiniLM-L6-v2",
+ help="Local SentenceTransformer model for memory retrieval")
+ parser.add_argument("--benchmarks", default="browsecompplus,swebench,bfcl,tau2",
+ help="Comma-separated benchmark slugs")
+ parser.add_argument("--output-dir", required=True)
+ parser.add_argument("--max-tokens", type=int, default=None, help="Max output tokens")
+ parser.add_argument("--reasoning-effort", default=None, help="Reasoning effort (low/medium/high)")
+ # ReasoningBank specific hyperparameters
+ parser.add_argument("--top-k-memories", type=int, default=1,
+ help="Number of memories to retrieve per query")
+ parser.add_argument("--max-memory-items", type=int, default=3,
+ help="Max memory items per induction")
+ args = parser.parse_args()
+ run_experiment(args)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/labs/AgentStream/exgentic/scripts/reasoning_bank/run_experiment.sh b/labs/AgentStream/exgentic/scripts/reasoning_bank/run_experiment.sh
new file mode 100644
index 00000000..0ebab8a9
--- /dev/null
+++ b/labs/AgentStream/exgentic/scripts/reasoning_bank/run_experiment.sh
@@ -0,0 +1,70 @@
+#!/bin/bash
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+set -e
+
+export OPENAI_API_BASE=""
+export OPENAI_API_KEY=""
+
+
+export CUDA_VISIBLE_DEVICES=4
+
+cd "$(dirname "$0")"
+
+SEED=44
+NUM_TASKS=50
+MODEL="openai/gpt-5.4"
+MAX_TOKENS="default"
+REASONING_EFFORT="default"
+MODE="interleaved" # isolated | sequential | interleaved
+
+# ReasoningBank specific hyperparameters
+TOP_K_MEMORIES=1
+MAX_MEMORY_ITEMS=3
+EMBEDDING_MODEL="all-MiniLM-L6-v2"
+
+OUTPUT_BASE="./outputs"
+MODEL_SHORT=$(echo $MODEL | sed 's|openai/||; s|azure/||; s|/|_|g')
+RUN_TAG="rb_${MODE}_s${SEED}_${MODEL_SHORT}_${MAX_TOKENS}_${REASONING_EFFORT}"
+
+SETTINGS_ARGS=""
+[ "$MAX_TOKENS" != "default" ] && SETTINGS_ARGS="$SETTINGS_ARGS --max-tokens $MAX_TOKENS"
+[ "$REASONING_EFFORT" != "default" ] && SETTINGS_ARGS="$SETTINGS_ARGS --reasoning-effort $REASONING_EFFORT"
+echo "Mode: ${MODE}"
+echo "ModelSettings args: ${SETTINGS_ARGS:-default (no overrides)}"
+echo "ReasoningBank: top_k=${TOP_K_MEMORIES} max_items=${MAX_MEMORY_ITEMS} embedding=${EMBEDDING_MODEL}"
+
+mkdir -p "$OUTPUT_BASE"
+
+ALL_BENCHMARKS="hle,bfcl,browsecompplus,appworld,swebench,tau2"
+if [ "$MODE" = "isolated" ]; then
+ for BENCH in swebench tau2 browsecompplus appworld hle bfcl; do
+ echo "=== Running isolated ${BENCH} ==="
+ uv run python run_experiment.py \
+ --mode isolated --seed $SEED --num-tasks $NUM_TASKS \
+ --model $MODEL \
+ --embedding-model $EMBEDDING_MODEL \
+ --top-k-memories $TOP_K_MEMORIES --max-memory-items $MAX_MEMORY_ITEMS \
+ $SETTINGS_ARGS \
+ --benchmarks $BENCH \
+ --output-dir ${OUTPUT_BASE}/${RUN_TAG}_${BENCH} \
+ 2>&1 | tee ${OUTPUT_BASE}/${RUN_TAG}_${BENCH}.log
+ done
+else
+ echo "=== Running ${MODE} (all benchmarks) ==="
+ uv run python run_experiment.py \
+ --mode $MODE --seed $SEED --num-tasks $NUM_TASKS \
+ --model $MODEL \
+ --embedding-model $EMBEDDING_MODEL \
+ --top-k-memories $TOP_K_MEMORIES --max-memory-items $MAX_MEMORY_ITEMS \
+ $SETTINGS_ARGS \
+ --benchmarks $ALL_BENCHMARKS \
+ --output-dir ${OUTPUT_BASE}/${RUN_TAG}_all \
+ 2>&1 | tee ${OUTPUT_BASE}/${RUN_TAG}_all.log
+fi
+
+echo ""
+echo "=== All benchmarks complete ==="
+echo "Run tag: ${RUN_TAG}"
+echo "Outputs in: ${OUTPUT_BASE}/${RUN_TAG}_*"
diff --git a/labs/AgentStream/exgentic/scripts/utils/task_ordering.py b/labs/AgentStream/exgentic/scripts/utils/task_ordering.py
new file mode 100644
index 00000000..6d944a9c
--- /dev/null
+++ b/labs/AgentStream/exgentic/scripts/utils/task_ordering.py
@@ -0,0 +1,210 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+"""
+Seed-controlled task selection and ordering for evaluation experiments.
+
+Guarantees:
+ - Same seed → same task set for every benchmark, regardless of mode.
+ - Within-benchmark task order is identical across isolated / sequential / interleaved.
+ - Interleaved only interleaves *between* benchmarks; within-benchmark order is preserved.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import random
+import sys
+from collections import deque
+from pathlib import Path
+from typing import Any
+
+# Allow importing exgentic from the repo source tree
+sys.path.insert(0, str(Path(__file__).resolve().parents[2] / "src"))
+
+from exgentic.interfaces.registry import load_benchmark
+
+
+# ──────────────────────────────────────────────────────────────
+# Public API
+# ──────────────────────────────────────────────────────────────
+
+def get_unified_task_order(
+ benchmark_configs: dict[str, dict[str, Any]],
+ num_tasks_per_benchmark: int,
+ seed: int,
+ mode: str,
+) -> list[tuple[str, str]]:
+ """Return a deterministic, mode-aware task ordering.
+
+ Parameters
+ ----------
+ benchmark_configs : dict
+ ``{slug: {"bm_kwargs": {...}, "agent_kwargs": {...}}}``
+ num_tasks_per_benchmark : int
+ How many tasks to select from each benchmark (e.g. 50).
+ seed : int
+ Ordering seed. Controls within-benchmark task order and
+ interleaved interleaving. Task *selection* is always fixed at
+ seed=42 so all experiments use the same task set.
+ mode : str
+ ``"isolated"`` | ``"sequential"`` | ``"interleaved"``.
+
+ Returns
+ -------
+ list of (benchmark_slug, task_id)
+ Ordered task sequence. For isolated/sequential the tasks are grouped
+ by benchmark (sorted alphabetically by slug). For interleaved the
+ tasks are interleaved across benchmarks while preserving
+ within-benchmark order.
+ """
+ # Always use seed=42 for task SELECTION (which tasks to pick),
+ # use the provided seed only for ORDERING (task sequence).
+ _SELECTION_SEED = 42
+ per_bm_tasks = _select_tasks(benchmark_configs, num_tasks_per_benchmark, _SELECTION_SEED)
+
+ # Re-shuffle within-benchmark order using the provided seed
+ if seed != _SELECTION_SEED:
+ for slug in per_bm_tasks:
+ order_seed = _derive_seed(seed, slug)
+ rng = random.Random(order_seed)
+ rng.shuffle(per_bm_tasks[slug])
+
+ if mode in ("isolated", "sequential"):
+ result: list[tuple[str, str]] = []
+ for slug in sorted(per_bm_tasks):
+ for tid in per_bm_tasks[slug]:
+ result.append((slug, tid))
+ return result
+
+ if mode == "interleaved":
+ return _interleave_preserving_order(per_bm_tasks, seed)
+
+ raise ValueError(f"Unknown mode: {mode!r}")
+
+
+def select_tasks_only(
+ benchmark_configs: dict[str, dict[str, Any]],
+ num_tasks_per_benchmark: int,
+ seed: int,
+) -> dict[str, list[str]]:
+ """Return the selected tasks per benchmark (no ordering applied)."""
+ return _select_tasks(benchmark_configs, num_tasks_per_benchmark, seed)
+
+
+# ──────────────────────────────────────────────────────────────
+# Internals
+# ──────────────────────────────────────────────────────────────
+
+def _select_tasks(
+ benchmark_configs: dict[str, dict[str, Any]],
+ num_tasks: int,
+ seed: int,
+) -> dict[str, list[str]]:
+ """Select *num_tasks* tasks per benchmark using per-benchmark derived seeds."""
+ per_bm: dict[str, list[str]] = {}
+ for slug in sorted(benchmark_configs):
+ bm_kwargs = benchmark_configs[slug].get("bm_kwargs", {})
+ bm = load_benchmark(slug)(**bm_kwargs)
+ evaluator = bm.get_evaluator()
+ try:
+ all_ids = [str(t) for t in evaluator.list_tasks()]
+ finally:
+ try:
+ evaluator.close()
+ except Exception:
+ pass
+ bm.close()
+
+ bm_seed = _derive_seed(seed, slug)
+ rng = random.Random(bm_seed)
+ rng.shuffle(all_ids)
+ per_bm[slug] = all_ids[:num_tasks]
+ return per_bm
+
+
+def _interleave_preserving_order(
+ per_bm_tasks: dict[str, list[str]],
+ seed: int,
+) -> list[tuple[str, str]]:
+ """Interleave tasks across benchmarks, preserving within-benchmark order.
+
+ At each step, randomly pick a non-empty benchmark queue and pop
+ its next task. This ensures the relative order within each
+ benchmark is the same as in isolated/sequential mode.
+ """
+ queues = {slug: deque(tasks) for slug, tasks in per_bm_tasks.items()}
+ rng = random.Random(seed)
+ result: list[tuple[str, str]] = []
+ while any(queues.values()):
+ available = sorted(s for s, q in queues.items() if q)
+ slug = rng.choice(available)
+ result.append((slug, queues[slug].popleft()))
+ return result
+
+
+def _derive_seed(master_seed: int, slug: str) -> int:
+ """Derive a deterministic per-benchmark seed from master seed + slug."""
+ h = hashlib.md5(f"{master_seed}_{slug}".encode()).hexdigest()
+ return int(h, 16) % (2**31)
+
+
+# ──────────────────────────────────────────────────────────────
+# Helpers
+# ──────────────────────────────────────────────────────────────
+
+def group_by_benchmark(
+ task_order: list[tuple[str, str]],
+) -> list[tuple[str, list[str]]]:
+ """Group a task order list into (slug, [task_ids]) preserving order."""
+ groups: list[tuple[str, list[str]]] = []
+ current_slug: str | None = None
+ current_ids: list[str] = []
+ for slug, tid in task_order:
+ if slug != current_slug:
+ if current_slug is not None:
+ groups.append((current_slug, current_ids))
+ current_slug = slug
+ current_ids = [tid]
+ else:
+ current_ids.append(tid)
+ if current_slug is not None:
+ groups.append((current_slug, current_ids))
+ return groups
+
+
+# ──────────────────────────────────────────────────────────────
+# Self-test
+# ──────────────────────────────────────────────────────────────
+
+if __name__ == "__main__":
+ # Quick sanity check without requiring benchmark data
+ print("=== task_ordering.py self-test ===\n")
+
+ # Simulate with fake data
+ fake_per_bm = {
+ "bfcl": ["b1", "b2", "b3", "b4", "b5"],
+ "tau2": ["t1", "t2", "t3", "t4", "t5"],
+ "browsecompplus": ["c1", "c2", "c3", "c4", "c5"],
+ }
+
+ # Test interleave preserving order
+ interleaved = _interleave_preserving_order(fake_per_bm, seed=42)
+ print("Interleaved interleave (seed=42):")
+ for slug, tid in interleaved:
+ print(f" {slug}: {tid}")
+
+ # Verify within-benchmark order is preserved
+ for slug in fake_per_bm:
+ original = fake_per_bm[slug]
+ fused = [tid for s, tid in interleaved if s == slug]
+ assert fused == original, f"{slug}: order changed! {original} → {fused}"
+ print(f" ✓ {slug} order preserved: {fused}")
+
+ # Verify different seeds produce different interleaving
+ interleaved2 = _interleave_preserving_order(fake_per_bm, seed=123)
+ order1 = [(s, t) for s, t in interleaved]
+ order2 = [(s, t) for s, t in interleaved2]
+ print(f"\n seed=42 vs seed=123 differ: {order1 != order2}")
+
+ print("\nAll checks passed.")
diff --git a/labs/AgentStream/exgentic/src/exgentic/__init__.py b/labs/AgentStream/exgentic/src/exgentic/__init__.py
new file mode 100644
index 00000000..285aa3dd
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/__init__.py
@@ -0,0 +1,86 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from __future__ import annotations
+
+from importlib import import_module
+from importlib.metadata import PackageNotFoundError
+from importlib.metadata import version as package_version
+from typing import Any
+
+try:
+ from ._version import version as __version__
+except ImportError:
+ try:
+ __version__ = package_version("exgentic")
+ except PackageNotFoundError:
+ __version__ = "0+unknown"
+
+from .environment.manager import EnvironmentManager, EnvType
+from .interfaces.registry import get_agent_entries, get_benchmark_entries
+
+_API_EXPORTS = {
+ "aggregate",
+ "evaluate",
+ "execute",
+ "list_agents",
+ "list_benchmarks",
+ "list_subsets",
+ "list_tasks",
+ "preview",
+ "results",
+ "status",
+}
+
+__all__ = [
+ "__version__",
+ "EnvironmentManager",
+ "EnvType",
+ "aggregate",
+ "evaluate",
+ "execute",
+ "list_agents",
+ "list_benchmarks",
+ "list_subsets",
+ "list_tasks",
+ "preview",
+ "results",
+ "status",
+]
+
+
+def _find_component_export(name: str):
+ matches = [
+ entry
+ for entries in (get_benchmark_entries(), get_agent_entries())
+ for entry in entries.values()
+ if entry.attr == name
+ ]
+ if not matches:
+ return None
+ if len(matches) > 1:
+ slugs = ", ".join(sorted(entry.slug_name for entry in matches))
+ raise AttributeError(f"Ambiguous exgentic export '{name}' found in registry entries: {slugs}.")
+ return matches[0]
+
+
+def __getattr__(name: str) -> Any:
+ if name in _API_EXPORTS:
+ value = getattr(import_module(".interfaces.lib.api", __name__), name)
+ globals()[name] = value
+ return value
+
+ entry = _find_component_export(name)
+ if entry is None:
+ raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
+ value = entry.load()
+ globals()[name] = value
+ return value
+
+
+def __dir__() -> list[str]:
+ # Keep dir() side-effect free.
+ # Some introspection libraries (e.g. freezegun) iterate over dir(module)
+ # and then call getattr() for each name. Exposing lazy registry exports here
+ # can trigger expensive imports during unrelated initialization paths.
+ return sorted(set(globals()) | _API_EXPORTS)
diff --git a/labs/AgentStream/exgentic/src/exgentic/adapters/__init__.py b/labs/AgentStream/exgentic/src/exgentic/adapters/__init__.py
new file mode 100644
index 00000000..367fec41
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/adapters/__init__.py
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
diff --git a/labs/AgentStream/exgentic/src/exgentic/adapters/actions/__init__.py b/labs/AgentStream/exgentic/src/exgentic/adapters/actions/__init__.py
new file mode 100644
index 00000000..ed18cc54
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/adapters/actions/__init__.py
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+__all__ = []
diff --git a/labs/AgentStream/exgentic/src/exgentic/adapters/actions/chat.py b/labs/AgentStream/exgentic/src/exgentic/adapters/actions/chat.py
new file mode 100644
index 00000000..0e1f5d86
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/adapters/actions/chat.py
@@ -0,0 +1,105 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+"""Chat/tool-call helpers for translating between Exgentic actions and chat payloads."""
+from __future__ import annotations
+
+from typing import Any, Optional
+
+from pydantic import BaseModel
+
+from ...core.actions import extract_argument
+from ...core.types import (
+ MessageObservation,
+ MessagePayload,
+ MultiObservation,
+ SingleAction,
+ SingleObservation,
+)
+
+
+class ChatActionContext:
+ """Helper to map actions to chat content/tool calls and back to observations."""
+
+ def __init__(self) -> None:
+ self.message_actions: dict[str, SingleAction] = {}
+ self.tool_actions: dict[str, SingleAction] = {}
+
+ @staticmethod
+ def action_to_tool_call_payload(action: SingleAction) -> dict[str, Any]:
+ arguments: Any = action.arguments
+ if isinstance(arguments, str):
+ arguments = {"error_parsing": arguments}
+ elif isinstance(arguments, BaseModel):
+ arguments = arguments.model_dump()
+ return {"name": action.name, "arguments": arguments, "id": action.id}
+
+ def actions_to_chat_components(self, actions: list[SingleAction]) -> tuple[Optional[str], list[dict[str, Any]]]:
+ content: Optional[str] = None
+ tool_calls: list[dict[str, Any]] = []
+ self.message_actions = {}
+ self.tool_actions = {}
+
+ for act in actions:
+ if act.name == "message":
+ self.message_actions[act.id] = act
+ msg_val = extract_argument(act.arguments, "content", None)
+ if msg_val is None:
+ try:
+ msg_val = str(act.arguments)
+ except Exception:
+ msg_val = None
+ if msg_val is not None:
+ if content is None:
+ content = ""
+ content += str(msg_val)
+ continue
+
+ self.tool_actions[act.id] = act
+ tool_calls.append(self.action_to_tool_call_payload(act))
+
+ return content, tool_calls
+
+ def actions_to_assistant_message(self, actions: list[SingleAction]) -> dict[str, Any]:
+ """Convert actions into an assistant message dict with content and tool_calls."""
+ content, tool_calls = self.actions_to_chat_components(actions)
+ message: dict[str, Any] = {"role": "assistant"}
+ if content is not None:
+ message["content"] = content
+ if tool_calls:
+ message["tool_calls"] = tool_calls
+ return message
+
+ def message_to_observation(self, message: Any) -> SingleObservation | MultiObservation:
+ # Support a list of messages (e.g., multiple tool responses)
+ if isinstance(message, list):
+ items = [self.message_to_observation(m) for m in message]
+ flat: list[SingleObservation] = []
+ for obs in items:
+ if isinstance(obs, MultiObservation):
+ flat.extend(obs.observations)
+ else:
+ flat.append(obs)
+ return MultiObservation(observations=flat)
+
+ if isinstance(message, dict):
+ role = message.get("role")
+ if role == "user":
+ acts = list(self.message_actions.values())
+ content = message.get("content") or ""
+ payload = MessagePayload(sender="user", message=content)
+ return MessageObservation(invoking_actions=acts, result=payload)
+ if role == "tool":
+ act = self.tool_actions.get(str(message.get("tool_call_id")))
+ return SingleObservation(
+ invoking_actions=([act] if act else []),
+ result=message.get("content"),
+ )
+ if "id" in message and "content" in message:
+ act = self.tool_actions.get(str(message.get("id")))
+ return SingleObservation(
+ invoking_actions=([act] if act else []),
+ result=message.get("content"),
+ )
+
+ return SingleObservation(invoking_actions=[], result=str(message))
diff --git a/labs/AgentStream/exgentic/src/exgentic/adapters/actions/functions.py b/labs/AgentStream/exgentic/src/exgentic/adapters/actions/functions.py
new file mode 100644
index 00000000..f730d0bb
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/adapters/actions/functions.py
@@ -0,0 +1,94 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+import inspect
+from typing import Any, Callable
+
+from pydantic import BaseModel
+from pydantic_core import PydanticUndefined
+
+from ...core.actions import build_action
+from ...core.types import ActionType, MultiObservation, SingleAction, SingleObservation
+
+
+def action_type_to_function(
+ action_type: "ActionType", internal_function: Callable[["SingleAction"], Any]
+) -> Callable[..., Any]:
+ def function(*args, **kwargs: Any) -> Any:
+ all_kwargs = bind_arguments(cls=action_type.arguments, args=args, kwargs=kwargs)
+ action = build_action(action_type, all_kwargs)
+ observation = internal_function(action)
+ if observation is None:
+ return None
+ if isinstance(observation, SingleObservation):
+ return observation.result
+ if isinstance(observation, MultiObservation):
+ return [obs.result for obs in observation.observations]
+ raise TypeError(f"Unexpected observation type: {type(observation).__name__}")
+
+ function.__name__ = action_type.name.replace(".", "__")
+
+ docstring_parts = [action_type.description or action_type.name.replace("_", " ")]
+
+ arguments_type = action_type.arguments
+ if not isinstance(arguments_type, type) or not issubclass(arguments_type, BaseModel):
+ raise TypeError(f"Action arguments must be a Pydantic BaseModel, got {arguments_type!r}")
+
+ params = []
+ annotations: dict[str, Any] = {}
+
+ fields = arguments_type.model_fields
+ if fields:
+ docstring_parts.extend(["", "Args:"])
+ for field_name, field_info in fields.items():
+ anno = field_info.annotation or Any
+
+ default = field_info.default
+ required = default is PydanticUndefined
+ if default is None:
+ required = False
+
+ param = inspect.Parameter(
+ name=field_name,
+ kind=inspect.Parameter.KEYWORD_ONLY,
+ annotation=anno,
+ default=(inspect._empty if required else default),
+ )
+ params.append(param)
+ annotations[field_name] = anno
+
+ # Description and Google-style formatting
+ desc = field_info.description or field_name.replace("_", " ")
+ type_name = getattr(anno, "__name__", None) or str(anno).replace("typing.", "")
+ docstring_parts.append(f" {field_name} ({type_name}): {desc}")
+
+ function.__signature__ = inspect.Signature(parameters=params)
+ function.__annotations__ = annotations
+ else:
+ function.__signature__ = inspect.Signature(parameters=[])
+ function.__annotations__ = {}
+
+ function.__annotations__["return"] = Any
+ function.__doc__ = "\n".join(docstring_parts)
+
+ return function
+
+
+def bind_arguments(cls: type[BaseModel], args: list[Any], kwargs: dict[str, Any]) -> dict[str, Any]:
+ """Bind positional args to Pydantic model fields by declaration order."""
+ field_names = list(cls.model_fields.keys())
+ if len(args) > len(field_names):
+ raise TypeError(
+ f"Too many positional arguments for {cls.__name__} "
+ f"(expected at most {len(field_names)}, got {len(args)})"
+ )
+
+ positional_names = field_names[: len(args)]
+ _duplicates = set(positional_names) & set(kwargs.keys())
+ if _duplicates:
+ dup_list = ", ".join(sorted(_duplicates))
+ raise TypeError(f"Multiple values for argument(s): {dup_list}")
+
+ bound = dict(zip(positional_names, args))
+ bound.update(kwargs)
+ return bound
diff --git a/labs/AgentStream/exgentic/src/exgentic/adapters/agents/__init__.py b/labs/AgentStream/exgentic/src/exgentic/adapters/agents/__init__.py
new file mode 100644
index 00000000..ed18cc54
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/adapters/agents/__init__.py
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+__all__ = []
diff --git a/labs/AgentStream/exgentic/src/exgentic/adapters/agents/code_agent.py b/labs/AgentStream/exgentic/src/exgentic/adapters/agents/code_agent.py
new file mode 100644
index 00000000..bf638121
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/adapters/agents/code_agent.py
@@ -0,0 +1,52 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from abc import abstractmethod
+from typing import Callable, List, Optional
+
+from ...core import Observation
+from ..actions.functions import action_type_to_function
+from .coordinator import AgentCoordinator, CoordinatedAgent
+
+
+class CodeAgentInstance(CoordinatedAgent, AgentCoordinator):
+ """Base class for code-based agents that inherits both roles."""
+
+ def __init__(self, session_id: str):
+ self.initial_observation: Optional[Observation] = None
+ # Initialize AgentCoordinator with self as the internal agent
+ AgentCoordinator.__init__(self, session_id, self)
+
+ def run(self, adapter) -> None:
+ """Implementation of CoordinatedAgent.run that converts actions to functions.
+
+ When the code agent calls one of the functions, what actually happens, is that the
+ AgentCoordinator.execute() method is called with the action. This places the action in an
+ action queue, creates a future for the result, and waits for it.
+
+ The AgentCordinator, which is running in a different thread, waits for an action in the
+ queue, fetches it and passes it to the benchmark environment. When the AgentCordinator receives
+ the coressponding observation, it places it in the result future.
+
+ This unblocks the CoordinateAgent, and cause the function to return the value.
+ The code agent then continues its run.
+
+ """
+ functions = []
+
+ for action_type in self.actions:
+ function = action_type_to_function(action_type, self.execute)
+ functions.append(function)
+
+ # Block until the environment delivers the initial observation via adapter.react()
+ self.initial_observation = adapter.get_observation()
+
+ try:
+ self.run_code_agent(functions)
+ finally:
+ self.execute(None) # Mark execution as done, by returning no action to the benchmark.
+
+ @abstractmethod
+ def run_code_agent(self, functions: List[Callable]) -> None:
+ """Subclasses implement their code agent logic here."""
+ pass
diff --git a/labs/AgentStream/exgentic/src/exgentic/adapters/agents/coordinator.py b/labs/AgentStream/exgentic/src/exgentic/adapters/agents/coordinator.py
new file mode 100644
index 00000000..ebd58c9d
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/adapters/agents/coordinator.py
@@ -0,0 +1,321 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+import contextvars
+import threading
+import time
+import traceback
+from abc import ABC, abstractmethod
+from typing import List, Optional
+
+from ...core.agent_instance import AgentInstance
+from ...core.types import (
+ Action,
+ MultiObservation,
+ Observation,
+ ParallelAction,
+ SingleAction,
+)
+
+
+class CoordinatedAgent(ABC):
+ """Internal agent that runs inside an AgentCoordinator.
+
+ The agent:
+ - receives observations via get_observation()
+ - sends actions via execute()
+ - signals termination by execute(None)
+ """
+
+ @abstractmethod
+ def run(self, adapter) -> None:
+ pass
+
+
+class AgentCoordinator(AgentInstance):
+ """Coordinates turn-based communication between threads.
+
+ - an environment thread (react)
+ - an internal agent thread (run / execute)
+ """
+
+ def __init__(
+ self,
+ session_id,
+ internal_agent: CoordinatedAgent,
+ accumulate_window_seconds: float | None = None,
+ ):
+ super().__init__(session_id)
+ self.internal_agent = internal_agent
+ self._accumulate_window_seconds = accumulate_window_seconds
+
+ self._condition = threading.Condition()
+ self._thread: Optional[threading.Thread] = None
+ self._started = False
+ self._closed = False
+ self._agent_error: Optional[BaseException] = None
+ self._agent_traceback: str | None = None
+
+ self._turn = 0
+ self._current_observation: Observation | None = None
+ self._agent_seen_turn = -1
+
+ self._pending_actions: List[Action | None] = []
+ self._last_actions: List[SingleAction] = []
+
+ def start(self, task, context, actions) -> None:
+ """Receive work payload and start the internal agent thread (once)."""
+ super().start(task, context, actions)
+ with self._condition:
+ if self._started:
+ raise RuntimeError("AgentCoordinator already started")
+ self._started = True
+ ctx = contextvars.copy_context()
+ self._thread = threading.Thread(
+ target=ctx.run,
+ args=(self._run_internal_agent,),
+ name=f"AgentCoordinator[{self.session_id}]",
+ daemon=False,
+ )
+ self._thread.start()
+
+ def _run_internal_agent(self) -> None:
+ """Entry point for the internal agent thread."""
+ try:
+ self.internal_agent.run(self)
+ except BaseException as exc:
+ with self._condition:
+ self._agent_error = exc
+ self._agent_traceback = traceback.format_exc()
+ self._closed = True
+ self._turn += 1
+ self._current_observation = None
+ self._pending_actions.clear()
+ self._condition.notify_all()
+ self.logger.exception("Internal agent crashed")
+ finally:
+ self.close()
+
+ def _raise_if_agent_failed(self) -> None:
+ if self._agent_error is not None:
+ if isinstance(self._agent_error, Exception):
+ tb = self._agent_traceback or ""
+ raise RuntimeError(f"{self._agent_error}\n\n{tb}") from self._agent_error
+ raise RuntimeError("Internal agent failed") from self._agent_error
+
+ def _flush_actions(self) -> Action | None:
+ """Combine pending actions into a single Action or ParallelAction."""
+ if not self._pending_actions:
+ return None
+ actions = self._pending_actions
+ self._pending_actions = []
+
+ if any(a is None for a in actions):
+ self._closed = True
+ return None
+
+ return actions[0] if len(actions) == 1 else ParallelAction(actions=actions)
+
+ def _remember_actions(self, action: Action | None) -> None:
+ if isinstance(action, Action):
+ actions = list(action.to_action_list())
+ if all(isinstance(act, SingleAction) for act in actions):
+ self._last_actions = actions
+ return
+ else:
+ self._last_actions = []
+ return
+ self._last_actions = []
+
+ def _rewire_observation(self, observation: Observation | None) -> Observation | None:
+ if observation is None or not self._last_actions:
+ return observation
+ if not isinstance(observation, Observation):
+ return observation
+
+ obs_list = observation.to_observation_list()
+ if not obs_list:
+ return observation
+
+ if len(obs_list) == 1 and len(self._last_actions) > 1:
+ obs = obs_list[0]
+ if not obs.invoking_actions:
+ obs.invoking_actions = list(self._last_actions)
+ return observation
+
+ used_ids = {act.id for obs in obs_list for act in obs.invoking_actions if isinstance(act, SingleAction)}
+ remaining = [act for act in self._last_actions if act.id not in used_ids]
+ for obs in obs_list:
+ if obs.invoking_actions:
+ continue
+ if not remaining:
+ break
+ obs.invoking_actions = [remaining.pop(0)]
+
+ if remaining:
+ self.logger.warning(
+ "Unassigned actions after rewiring observations (actions=%s, observations=%s)",
+ len(self._last_actions),
+ len(obs_list),
+ )
+ return observation
+
+ def _select_observation_for_action(
+ self, action: Action | None, observation: Observation | None
+ ) -> Observation | None:
+ if observation is None:
+ return observation
+ if not isinstance(observation, Observation):
+ return observation
+ if not isinstance(action, SingleAction):
+ return observation
+
+ obs_list = observation.to_observation_list()
+ if not obs_list:
+ return observation
+
+ matched = [
+ obs
+ for obs in obs_list
+ if any(isinstance(inv, SingleAction) and inv.id == action.id for inv in obs.invoking_actions)
+ ]
+ if matched:
+ if len(matched) == 1:
+ return matched[0]
+ return MultiObservation(observations=matched)
+
+ self.logger.warning(
+ "No matching observation for action id=%s (observations=%s)",
+ action.id,
+ len(obs_list),
+ )
+ return observation
+
+ def _publish_observation(self, observation: Observation | None) -> None:
+ self.logger.info("Publishing observation (turn=%s): %s", self._turn + 1, observation)
+ observation = self._rewire_observation(observation)
+ self._last_actions = []
+ self._current_observation = observation
+ self._turn += 1
+ self._condition.notify_all()
+
+ def _wait_for_pending_actions(self) -> bool:
+ while not self._closed and not self._pending_actions:
+ self._condition.wait()
+ self._raise_if_agent_failed()
+ return not self._closed
+
+ def _accumulate_pending_actions(self) -> None:
+ if not self._accumulate_window_seconds:
+ return
+ if any(a is None for a in self._pending_actions):
+ return
+ deadline = time.monotonic() + self._accumulate_window_seconds
+ while not self._closed:
+ remaining = deadline - time.monotonic()
+ if remaining <= 0:
+ return
+ if any(a is None for a in self._pending_actions):
+ return
+ self._condition.wait(timeout=remaining)
+
+ def get_observation(self) -> Observation | None:
+ """Block until a new observation is available or the coordinator closes."""
+ with self._condition:
+ self._raise_if_agent_failed()
+ if self._closed:
+ return None
+
+ # Wait until a new observation is published by the environment thread.
+ # We also block if _current_observation is still None (initial call).
+ while not self._closed and (self._turn <= self._agent_seen_turn or self._current_observation is None):
+ self._condition.wait()
+
+ self._raise_if_agent_failed()
+ if self._closed:
+ return None
+
+ self._agent_seen_turn = self._turn
+ self.logger.info(
+ "Delivered observation (turn=%s): %s",
+ self._turn,
+ self._current_observation,
+ )
+ return self._current_observation
+
+ def execute(self, action: Action | None) -> Observation | None:
+ """Publish an action for the current turn.
+
+ If action is None, signals agent termination.
+ """
+ with self._condition:
+ self._raise_if_agent_failed()
+ if self._closed:
+ if action is not None:
+ raise RuntimeError("execute() called after close()")
+ return None
+
+ self.logger.info("Received action (turn=%s): %s", self._turn, action)
+ self._pending_actions.append(action)
+ self._condition.notify_all()
+
+ if action is None:
+ self._closed = True
+ return None
+
+ my_turn = self._turn
+ while not self._closed and self._turn == my_turn:
+ self._condition.wait()
+
+ self._raise_if_agent_failed()
+ if self._closed:
+ return None
+
+ self._agent_seen_turn = self._turn
+ self.logger.info(
+ "Delivered observation after action (turn=%s): %s",
+ self._turn,
+ self._current_observation,
+ )
+ return self._select_observation_for_action(action, self._current_observation)
+
+ def react(self, observation: Observation | None) -> Action | None:
+ """Publish an observation and wait for the agent's action.
+
+ If observation is None, signals environment termination.
+ """
+ with self._condition:
+ self._raise_if_agent_failed()
+ if self._closed:
+ if observation is not None:
+ raise RuntimeError("react() called after close()")
+ return None
+
+ self._publish_observation(observation)
+
+ if observation is None:
+ self._closed = True
+ return None
+
+ if not self._wait_for_pending_actions():
+ return None
+
+ self._accumulate_pending_actions()
+
+ action = self._flush_actions()
+ self._remember_actions(action)
+ return action
+
+ def close(self) -> None:
+ """Close the coordinator and wait for the agent thread to exit."""
+ with self._condition:
+ if not self._closed:
+ self._closed = True
+ self._turn += 1
+ self._current_observation = None
+ self._pending_actions.clear()
+ self._condition.notify_all()
+ t = self._thread
+
+ if t and t.is_alive() and t is not threading.current_thread():
+ t.join()
diff --git a/labs/AgentStream/exgentic/src/exgentic/adapters/agents/mcp_agent.py b/labs/AgentStream/exgentic/src/exgentic/adapters/agents/mcp_agent.py
new file mode 100644
index 00000000..424b20f7
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/adapters/agents/mcp_agent.py
@@ -0,0 +1,104 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+import abc
+from abc import abstractmethod
+from typing import Any, Callable, List, Optional
+
+from ..actions.functions import action_type_to_function
+from .code_agent import CodeAgentInstance
+from .mcp_server import MCPServer
+
+
+class MCPAgentInstance(CodeAgentInstance, abc.ABC):
+ """Sync-first base class.
+
+ - run_code_agent(): sync
+ - run_mcp_agent(): ABSTRACT SYNC (subclass decides implementation strategy)
+ """
+
+ def __init__(self, session_id: str):
+ self._mcp_server: Optional[MCPServer] = None
+ super().__init__(session_id)
+
+ def run(self, adapter) -> None:
+ functions: List[Callable[..., Any]] = []
+ for action_type in self.actions:
+ if action_type.is_finish:
+ function = action_type_to_function(action_type, self._submit_finish_action)
+ else:
+ function = action_type_to_function(action_type, self.execute)
+ functions.append(function)
+
+ self.initial_observation = adapter.get_observation()
+
+ try:
+ self.run_code_agent(functions)
+ finally:
+ self.execute(None)
+
+ def run_code_agent(self, functions: List[Callable[..., Any]]) -> Any:
+ """Fully synchronous entrypoint."""
+ self.logger.info("Starting MCP server for agent tools")
+ server = MCPServer(
+ tools=functions,
+ log_dir=self.paths.agent_dir,
+ logger=self.logger,
+ stringify_empty_output=self._stringify_empty_output(),
+ )
+ self._mcp_server = server
+ self.mcp = server.mcp
+ started = False
+ try:
+ with server:
+ started = True
+ self.logger.info(
+ "MCP server ready at http://%s:%s/mcp",
+ server.connect_host,
+ server.port,
+ )
+ return self.run_mcp_agent(server.connect_host, server.port)
+ finally:
+ if started:
+ self.logger.info("MCP server stopped")
+
+ def _stringify_empty_output(self) -> bool:
+ return False
+
+ def _submit_finish_action(self, action) -> None:
+ with self._condition:
+ self._raise_if_agent_failed()
+ if self._closed:
+ if action is not None:
+ raise RuntimeError("execute() called after close()")
+ return
+
+ self.logger.info("Received finish action (turn=%s): %s", self._turn, action)
+ self._pending_actions.append(action)
+ self._condition.notify_all()
+ return
+
+ def close_mcp_agent(self) -> None:
+ self.close()
+
+ def close(self) -> None:
+ super().close()
+ if self._mcp_server is not None:
+ self.logger.info("Stopping MCP server")
+ self._mcp_server.stop(raise_on_timeout=False)
+
+ @abstractmethod
+ def run_mcp_agent(self, mcp_host: str, mcp_port: int) -> Any:
+ """ABSTRACT SYNC.
+
+ Subclass may implement:
+ - purely sync logic, OR
+ - a sync wrapper over an async core (via run_sync, etc.)
+ """
+ ...
+
+
+class MCPAgent(MCPAgentInstance, abc.ABC):
+ """Backwards-compatible alias for MCPAgentInstance."""
+
+ pass
diff --git a/labs/AgentStream/exgentic/src/exgentic/adapters/agents/mcp_server.py b/labs/AgentStream/exgentic/src/exgentic/adapters/agents/mcp_server.py
new file mode 100644
index 00000000..1a97ebb6
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/adapters/agents/mcp_server.py
@@ -0,0 +1,351 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+import functools
+import inspect
+import logging
+import socket
+import threading
+import time
+from typing import Any, Callable, Iterable, Optional
+
+import uvicorn
+from mcp.client.session import ClientSession
+from mcp.client.streamable_http import streamable_http_client
+from mcp.server.fastmcp.server import FastMCP
+from pydantic import BaseModel
+
+from ...observers.logging import (
+ configure_library_file_logging,
+ configure_uvicorn_file_logging,
+ get_logger,
+)
+from ...utils.sync import run_sync
+
+
+class MCPServerConfig(BaseModel):
+ http_timeout_seconds: float | None = None # None means no timeout
+ sse_read_timeout_seconds: float | None = None # None means no timeout
+ http_connect_timeout_seconds: float | None = None # None means no timeout
+ headers: dict[str, str] | None = None
+ terminate_on_close: bool = True
+
+
+class MCPServer:
+ _MAX_SAFE_SCHEMA_INT = 2_147_483_647
+
+ def __init__(
+ self,
+ mcp: FastMCP | None = None,
+ *,
+ host: str | None = None,
+ port: int | None = None,
+ tools: Iterable[Callable[..., Any]] | None = None,
+ log_dir,
+ logger: logging.Logger,
+ stringify_empty_output: bool = False,
+ ) -> None:
+ self._mcp = mcp or self._build_fastmcp()
+ self._host = host or "0.0.0.0"
+ self._port = port
+ self._log_dir = log_dir
+ self._logger = logger
+ self._stringify_empty_output = stringify_empty_output
+ self._mcp_log_dir = self._log_dir / "mcp"
+ self._mcp_log_dir.mkdir(parents=True, exist_ok=True)
+ self._server_logger = get_logger(
+ f"MCPServer_{id(self)}",
+ str(self._mcp_log_dir / "server.log"),
+ )
+ ts = self._mcp.settings.transport_security
+ ts.allowed_hosts = [
+ *ts.allowed_hosts,
+ "host.containers.internal:*",
+ "host.docker.internal:*",
+ ]
+ ts.allowed_origins = [
+ *ts.allowed_origins,
+ "http://host.containers.internal:*",
+ "http://host.docker.internal:*",
+ ]
+
+ tool_names: list[str] = []
+ if tools:
+ for fn in tools:
+ tool_fn = self._wrap_tool(fn) if self._stringify_empty_output else fn
+ tool = self._mcp._tool_manager.add_tool(tool_fn)
+ tool.parameters = self._clamp_schema_ints(tool.parameters)
+ tool_names.append(fn.__name__)
+ self._log_tool_summary(tool_names)
+
+ if self._logger is not None:
+ self._logger.info("MCP logs at %s", self._mcp_log_dir)
+
+ self._server: Optional[uvicorn.Server] = None
+ self._thread: Optional[threading.Thread] = None
+ self._sock: Optional[socket.socket] = None
+ self._started = threading.Event()
+
+ @classmethod
+ def _clamp_schema_ints(cls, obj: Any) -> Any:
+ if isinstance(obj, dict):
+ return {k: cls._clamp_schema_ints(v) for k, v in obj.items()}
+ if isinstance(obj, list):
+ return [cls._clamp_schema_ints(v) for v in obj]
+ if isinstance(obj, int) and not isinstance(obj, bool):
+ if obj > cls._MAX_SAFE_SCHEMA_INT:
+ return cls._MAX_SAFE_SCHEMA_INT
+ if isinstance(obj, float):
+ if obj > cls._MAX_SAFE_SCHEMA_INT:
+ return float(cls._MAX_SAFE_SCHEMA_INT)
+ return obj
+
+ @staticmethod
+ def stringify_empty_output(result: Any) -> Any:
+ if result is None:
+ return "null"
+ if result == []:
+ return "[]"
+ return result
+
+ @classmethod
+ def _wrap_tool(cls, fn: Callable[..., Any]) -> Callable[..., Any]:
+ signature = inspect.signature(fn)
+
+ if inspect.iscoroutinefunction(fn):
+
+ @functools.wraps(fn)
+ async def wrapper(*args: Any, **kwargs: Any) -> Any:
+ result = await fn(*args, **kwargs)
+ return cls.stringify_empty_output(result)
+
+ else:
+
+ @functools.wraps(fn)
+ def wrapper(*args: Any, **kwargs: Any) -> Any:
+ result = fn(*args, **kwargs)
+ return cls.stringify_empty_output(result)
+
+ wrapper.__signature__ = signature # type: ignore[attr-defined]
+ return wrapper
+
+ def _build_fastmcp(self) -> FastMCP:
+ root = logging.getLogger()
+ null_handler: logging.Handler | None = None
+ if not root.handlers:
+ null_handler = logging.NullHandler()
+ root.addHandler(null_handler)
+ try:
+ return FastMCP()
+ finally:
+ if null_handler is not None:
+ try:
+ root.removeHandler(null_handler)
+ except Exception:
+ pass
+
+ @property
+ def server(self) -> Optional[uvicorn.Server]:
+ return self._server
+
+ @property
+ def thread(self) -> Optional[threading.Thread]:
+ return self._thread
+
+ @property
+ def mcp(self) -> FastMCP:
+ return self._mcp
+
+ @property
+ def host(self) -> str:
+ return self._host
+
+ @property
+ def connect_host(self) -> str:
+ # Use 127.0.0.1 for connecting, even if server binds to 0.0.0.0.
+ return "127.0.0.1" if self._host == "0.0.0.0" else self._host
+
+ @property
+ def port(self) -> int:
+ if self._port is None:
+ raise RuntimeError("MCP server port not assigned yet.")
+ return self._port
+
+ def start(
+ self,
+ timeout: float = 60.0,
+ *,
+ tcp_timeout: float = 60.0,
+ ping_timeout: float = 60.0,
+ ) -> None:
+ if self._thread and self._thread.is_alive():
+ return
+
+ self._ensure_socket()
+
+ self._started.clear()
+ t = threading.Thread(
+ target=self._thread_entry,
+ name=f"mcp-native:{self._port}",
+ daemon=True,
+ )
+ self._thread = t
+ t.start()
+
+ if not self._started.wait(timeout=timeout):
+ raise RuntimeError("MCP uvicorn thread did not signal startup within timeout.")
+
+ try:
+ started_at = time.time()
+ # Use 127.0.0.1 for connecting, even if server binds to 0.0.0.0
+ connect_host = "127.0.0.1" if self._host == "0.0.0.0" else self._host
+ wait_for_tcp(connect_host, self.port, timeout=tcp_timeout)
+ tcp_elapsed = time.time() - started_at
+ ping_started = time.time()
+ run_sync(
+ wait_for_mcp_ping_async(connect_host, self.port, timeout=ping_timeout),
+ timeout=ping_timeout + 5.0,
+ )
+ ping_elapsed = time.time() - ping_started
+ self._server_logger.info(
+ "MCP readiness OK (tcp=%.2fs, ping=%.2fs)",
+ tcp_elapsed,
+ ping_elapsed,
+ )
+ except BaseException as exc:
+ self.stop(error=exc, raise_on_timeout=False)
+ raise
+
+ def __enter__(self) -> "MCPServer":
+ self.start()
+ return self
+
+ def __exit__(self, exc_type, exc, tb) -> None:
+ self.stop(error=exc, raise_on_timeout=True)
+
+ def stop(
+ self,
+ timeout: float = 60.0,
+ *,
+ error: BaseException | None = None,
+ raise_on_timeout: bool = True,
+ ) -> None:
+ if self._server is not None:
+ self._server.should_exit = True
+
+ t = self._thread
+ if t and t.is_alive():
+ t.join(timeout=timeout)
+ if t.is_alive():
+ message = "MCP uvicorn server thread did not exit cleanly"
+ if raise_on_timeout and error is None:
+ raise RuntimeError(message)
+ self._server_logger.warning(message)
+ if self._sock is not None:
+ try:
+ self._sock.close()
+ finally:
+ self._sock = None
+
+ def _thread_entry(self) -> None:
+ thread_id = threading.get_ident()
+ cleanup_uvicorn = configure_uvicorn_file_logging(
+ self._mcp_log_dir / "uvicorn.log",
+ thread_id=thread_id,
+ )
+ cleanup_mcp = configure_library_file_logging(
+ self._mcp_log_dir / "server.log",
+ logger_names=["mcp", "mcp.server", "mcp.client"],
+ thread_id=thread_id,
+ )
+
+ server = self._build_server()
+
+ self._server_logger.info(
+ "Starting MCP server on %s:%s (path=%s)",
+ self._host,
+ self.port,
+ self._mcp.settings.streamable_http_path,
+ )
+ self._started.set()
+
+ try:
+ sock = self._sock
+ if sock is not None:
+ self._server_logger.info("Using pre-bound socket on %s:%s", self._host, self.port)
+ server.run(sockets=[sock])
+ else:
+ server.run()
+ finally:
+ self._server_logger.info("MCP server stopped")
+ cleanup_mcp()
+ cleanup_uvicorn()
+
+ def _ensure_socket(self) -> None:
+ if self._sock is not None:
+ return
+ sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+ try:
+ sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+ sock.bind((self._host, self._port or 0))
+ sock.listen(1)
+ except Exception:
+ sock.close()
+ raise
+ self._sock = sock
+ self._port = int(sock.getsockname()[1])
+
+ def _build_server(self) -> uvicorn.Server:
+ app = self._mcp.streamable_http_app()
+ config = uvicorn.Config(app, host=self._host, port=self.port, log_config=None)
+ server = uvicorn.Server(config)
+ self._server = server
+ return server
+
+ def _log_tool_summary(self, tool_names: list[str]) -> None:
+ count = len(tool_names)
+ if count == 0:
+ self._server_logger.info("Registered 0 MCP tools")
+ return
+ preview = ", ".join(tool_names[:6])
+ if count > 6:
+ preview = f"{preview}, +{count - 6} more"
+ self._server_logger.info("Registered %s MCP tool(s): %s", count, preview)
+
+
+def wait_for_tcp(host: str, port: int, timeout: float = 60.0) -> None:
+ deadline = time.time() + timeout
+ last_err: Optional[BaseException] = None
+ while time.time() < deadline:
+ try:
+ with socket.create_connection((host, port), timeout=0.5):
+ return
+ except Exception as e:
+ last_err = e
+ time.sleep(0.1)
+ raise TimeoutError(f"Server at {host}:{port} did not open TCP port within {timeout}s. " f"Last error: {last_err!r}")
+
+
+async def wait_for_mcp_ping_async(host: str, port: int, timeout: float = 60.0) -> None:
+ deadline = time.time() + timeout
+ last_err: Optional[BaseException] = None
+ url = f"http://{host}:{port}/mcp"
+
+ import asyncio # local import to keep module sync-first
+
+ while time.time() < deadline:
+ try:
+ async with streamable_http_client(url) as (
+ read_stream,
+ write_stream,
+ _,
+ ):
+ async with ClientSession(read_stream, write_stream) as session:
+ await session.initialize()
+ await session.send_ping()
+ return
+ except Exception as e:
+ last_err = e
+ await asyncio.sleep(0.1)
+
+ raise TimeoutError(f"MCP ping at {host}:{port} failed within {timeout}s. " f"Last error: {last_err!r}")
diff --git a/labs/AgentStream/exgentic/src/exgentic/adapters/executors/__init__.py b/labs/AgentStream/exgentic/src/exgentic/adapters/executors/__init__.py
new file mode 100644
index 00000000..ed18cc54
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/adapters/executors/__init__.py
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+__all__ = []
diff --git a/labs/AgentStream/exgentic/src/exgentic/adapters/executors/proxy.py b/labs/AgentStream/exgentic/src/exgentic/adapters/executors/proxy.py
new file mode 100644
index 00000000..e21e291b
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/adapters/executors/proxy.py
@@ -0,0 +1,199 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from argparse import Action
+from queue import Queue
+from threading import Event, Lock, Semaphore
+from typing import Any, Generic, Optional, TypeVar
+
+from ...core.session import Session
+from ...core.types import Observation
+
+DONE = object()
+
+
+class BaseProxySession(Session, ABC):
+ """Generic queue-based proxy session.
+
+ Implements a simple rendezvous between an external driver (agent/framework)
+ and a foreign environment adapter via two blocking queues.
+ """
+
+ def __init__(self):
+ super().__init__()
+ self.completed = False
+ self.step_count = 0
+ self._to_agent: Queue = Queue()
+ self._from_agent: Queue = Queue()
+ self._last_observation: Optional[Any] = None
+
+ def step(self, action: Action) -> Optional[Observation]:
+ self.step_count += 1
+ self._from_agent.put(action)
+ next_obs = self._to_agent.get()
+ return next_obs
+
+ def start(self) -> Optional[Observation]:
+ result = self._to_agent.get()
+ return result
+
+ def done(self) -> bool:
+ result = self.completed
+ return result
+
+ def score(self) -> dict:
+ return {
+ "success": self.completed,
+ "steps": self.step_count,
+ "score": max(0.0, 1.0 - (self.step_count - 1) * 0.1),
+ }
+
+ def close(self):
+ self.completed = True
+ try:
+ self._from_agent.put_nowait(DONE)
+ except Exception:
+ pass
+ try:
+ self._to_agent.put_nowait(DONE)
+ except Exception:
+ pass
+
+ # --- Hooks for subclasses ---
+ def put_observation(self, obs: Any) -> None:
+ self._last_observation = obs
+ self._to_agent.put(obs)
+
+ def wait_for_action(self) -> Optional[Any]:
+ item = self._from_agent.get()
+ if item is DONE:
+ self.completed = True
+ return None
+ return item
+
+
+_PAIRING_SEMAPHORE: Semaphore = Semaphore(1) # Only 1 session can be staged at a time
+_PAIRING_LOCK: Lock = Lock()
+_CURRENT_SESSION: Optional[PairableProxySession] = None
+
+
+class PairableProxySession(BaseProxySession):
+ """Proxy session that can be staged and paired with a proxy agent automatically."""
+
+ def __init__(self):
+ super().__init__()
+ self._paired_event: Event = Event()
+
+ # Pairing API
+ def stage_for_pairing(self) -> None:
+ _PAIRING_SEMAPHORE.acquire() # Blocks until slot available
+ with _PAIRING_LOCK:
+ global _CURRENT_SESSION
+ _CURRENT_SESSION = self
+
+ def _mark_paired(self) -> None:
+ self._paired_event.set()
+
+ def waiting_for_pairing(self):
+ return _CURRENT_SESSION == self and not self._paired_event.is_set()
+
+ def unstage_for_pairing(self):
+ if self.waiting_for_pairing():
+ with _PAIRING_LOCK:
+ global _CURRENT_SESSION
+ assert _CURRENT_SESSION == self
+ _CURRENT_SESSION = None
+ _PAIRING_SEMAPHORE.release()
+
+ def pair_to_agent(self, timeout: Optional[float] = 10.0) -> None:
+ ok = self._paired_event.wait(timeout=timeout)
+ if not ok:
+ raise RuntimeError("Timed out waiting for proxy agent to pair with session")
+
+ # Gate helpers
+ @classmethod
+ def block_pairing(cls) -> None:
+ pass # No longer needed with lock-based approach
+
+ @classmethod
+ def allow_pairing(cls) -> None:
+ pass # No longer needed with lock-based approach
+
+ @classmethod
+ def pairing_allowed(cls) -> bool:
+ return True # Always allowed with lock-based approach
+
+
+SessionT = TypeVar("SessionT", bound=BaseProxySession)
+
+
+class BaseProxyAgent(ABC, Generic[SessionT]):
+ """Base mixin providing generic step handling between a proxy session and an external environment.
+
+ Uses core terms (session, observation, action).
+
+ Adapters should call `handle_observation(observation, state)` from their
+ environment-specific entrypoint.
+ """
+
+ def _ensure_session(self, state: Optional[SessionT], observation: Any) -> SessionT:
+ if state is None:
+ return self.create_session(observation)
+ self.update_session_observation(state, observation)
+ return state
+
+ def handle_observation(self, observation: Any, state: Optional[SessionT]):
+ """Generic step handler: ensure session, wait for action, translate response.
+
+ - observation: an environment-specific observation object
+ - state: the proxy session instance (or None for a new session)
+ Returns (environment-specific response, new_state).
+ """
+ session = self._ensure_session(state, observation)
+ action = session.wait_for_action()
+ response_obj, new_state = self.action_to_response(action, observation, session)
+ return response_obj, new_state
+
+ # --- Subclass hooks ---
+ @abstractmethod
+ def create_session(self, first_observation: Any) -> SessionT:
+ pass
+
+ @abstractmethod
+ def update_session_observation(self, session: SessionT, observation: Any) -> None:
+ pass
+
+ @abstractmethod
+ def action_to_response(self, action: Any, observation: Any, session: SessionT):
+ pass
+
+
+class PairableProxyAgent(BaseProxyAgent[SessionT]):
+ """Proxy agent that adopts the currently staged PairableProxySession."""
+
+ def adopt_staged_session(self) -> SessionT:
+ with _PAIRING_LOCK:
+ global _CURRENT_SESSION
+ sess = _CURRENT_SESSION
+ if sess is None:
+ raise RuntimeError("No staged session available for pairing")
+ _CURRENT_SESSION = None
+ sess._mark_paired() # type: ignore[attr-defined]
+ _PAIRING_SEMAPHORE.release() # Release slot for next session
+ return sess # type: ignore[return-value]
+
+ # Gate helpers
+ @classmethod
+ def block_pairing(cls) -> None:
+ PairableProxySession.block_pairing()
+
+ @classmethod
+ def allow_pairing(cls) -> None:
+ PairableProxySession.allow_pairing()
+
+ @classmethod
+ def pairing_allowed(cls) -> bool:
+ return PairableProxySession.pairing_allowed()
diff --git a/labs/AgentStream/exgentic/src/exgentic/adapters/runners/__init__.py b/labs/AgentStream/exgentic/src/exgentic/adapters/runners/__init__.py
new file mode 100644
index 00000000..84d0360b
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/adapters/runners/__init__.py
@@ -0,0 +1,125 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+"""Runner & Transport abstractions for running objects in different isolation levels.
+
+Runners wrap any object and control where it executes:
+
+- ``direct`` — same thread, no isolation
+- ``thread`` — separate thread, queue-based communication
+- ``process`` — separate process, pipe-based communication with cloudpickle
+- ``service`` — HTTP service in a background thread
+- ``docker`` — HTTP service inside a Docker container
+- ``venv`` — HTTP service in an isolated uv virtual environment
+
+Usage::
+
+ calc = with_runner(Calculator, runner="thread", value=10)
+"""
+
+from __future__ import annotations
+
+from typing import Any, Literal
+
+from .direct import DirectTransport
+from .transport import ObjectHost, ObjectProxy, Transport
+
+RunnerName = Literal["direct", "thread", "process", "service", "docker", "venv"]
+
+
+def _resolve_cls(cls: type | str) -> type:
+ """Resolve a ``"module:qualname"`` string to the actual class."""
+ if isinstance(cls, type):
+ return cls
+ module_path, qualname = cls.rsplit(":", 1)
+ import importlib
+
+ mod = importlib.import_module(module_path)
+ obj = mod
+ for attr in qualname.split("."):
+ obj = getattr(obj, attr)
+ return obj # type: ignore[return-value]
+
+
+def with_runner(cls: type | str, *args: Any, runner: RunnerName = "direct", **kwargs: Any) -> Any:
+ """Create an instance of *cls* running in the specified isolation level.
+
+ *cls* may be a class or a ``"module:qualname"`` string. String
+ references are resolved lazily — for ``venv`` and ``docker`` runners
+ the string is forwarded directly so heavy imports never happen on the
+ host.
+
+ Returns an ``ObjectProxy`` that transparently forwards all
+ attribute access and method calls to the real object.
+ """
+ if runner == "direct":
+ cls = _resolve_cls(cls)
+ return ObjectProxy(DirectTransport(cls(*args, **kwargs)))
+
+ if runner == "thread":
+ from .thread import ThreadTransport
+
+ cls = _resolve_cls(cls)
+ t = ThreadTransport(cls, *args, **kwargs)
+ t.start()
+ return ObjectProxy(t)
+
+ if runner == "process":
+ from .process import PipeTransport
+
+ cls = _resolve_cls(cls)
+ t = PipeTransport(cls, *args, **kwargs)
+ t.start()
+ return ObjectProxy(t)
+
+ if runner == "service":
+ from .service import ServiceRunner
+
+ cls = _resolve_cls(cls)
+ return ServiceRunner(cls, *args, **kwargs).start()
+
+ if runner == "docker":
+ from .docker import DockerRunner
+
+ docker_kw = {}
+ for key in (
+ "env_name",
+ "module_path",
+ "image",
+ "dockerfile",
+ "port",
+ "docker_args",
+ "dependencies",
+ "docker_socket",
+ "volumes",
+ ):
+ if key in kwargs:
+ docker_kw[key] = kwargs.pop(key)
+ return DockerRunner(cls, *args, **docker_kw, **kwargs).start()
+
+ if runner == "venv":
+ from .venv import VenvRunner
+
+ venv_kw = {}
+ for key in (
+ "env_name",
+ "module_path",
+ "port",
+ "dependencies",
+ "health_timeout",
+ ):
+ if key in kwargs:
+ venv_kw[key] = kwargs.pop(key)
+ return VenvRunner(cls, *args, **venv_kw, **kwargs).start()
+
+ raise ValueError(f"Unknown runner: {runner!r}")
+
+
+__all__ = [
+ "DirectTransport",
+ "ObjectHost",
+ "ObjectProxy",
+ "RunnerName",
+ "Transport",
+ "with_runner",
+]
diff --git a/labs/AgentStream/exgentic/src/exgentic/adapters/runners/_utils.py b/labs/AgentStream/exgentic/src/exgentic/adapters/runners/_utils.py
new file mode 100644
index 00000000..07dac32f
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/adapters/runners/_utils.py
@@ -0,0 +1,138 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+"""Shared utilities for runner implementations."""
+
+from __future__ import annotations
+
+import base64
+import json
+import socket
+from pathlib import Path
+from typing import Any
+
+
+def find_project_root() -> Path:
+ """Return the project root directory.
+
+ Walks up from the exgentic package looking for a ``pyproject.toml``.
+ When none is found (e.g. ``uv tool install exgentic``), falls back
+ to ``~/.exgentic/`` so that benchmark venvs and caches still have a
+ stable home directory.
+ """
+ for parent in Path(__file__).resolve().parents:
+ if (parent / "pyproject.toml").exists():
+ return parent
+ fallback = Path.home() / ".exgentic"
+ fallback.mkdir(parents=True, exist_ok=True)
+ return fallback
+
+
+def find_free_port() -> int:
+ """Return an unused TCP port on localhost."""
+ with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+ s.bind(("", 0))
+ return s.getsockname()[1]
+
+
+def serialize_kwargs(kwargs: dict[str, Any]) -> tuple[str, str]:
+ """Serialize kwargs for the ``exgentic serve`` CLI.
+
+ Returns ``(flag, value)`` — either ``("--kwargs", json_str)``
+ or ``("--kwargs-b64", pickled_b64)`` for non-JSON-serializable values.
+ """
+ try:
+ return "--kwargs", json.dumps(kwargs)
+ except TypeError:
+ import cloudpickle as cp
+
+ return "--kwargs-b64", base64.b64encode(cp.dumps(kwargs)).decode("ascii")
+
+
+_SYSTEM_ENV_BLOCKLIST = frozenset(
+ {
+ "PATH",
+ "HOME",
+ "USER",
+ "SHELL",
+ "HOSTNAME",
+ "LANG",
+ "TERM",
+ "PWD",
+ "OLDPWD",
+ "SHLVL",
+ "_",
+ "TMPDIR",
+ "VIRTUAL_ENV",
+ "CONDA_DEFAULT_ENV",
+ "CONDA_PREFIX",
+ }
+)
+_PREFIX_BLOCKLIST = ("VSCODE_", "UV_", "PIP_")
+
+
+def prepare_subprocess_env() -> dict[str, str]:
+ """Build a filtered env dict for subprocess runners (venv, docker).
+
+ Forwards API tokens and user config while excluding system-level
+ vars, IDE noise, and Python-path-manager prefixes that could
+ conflict with the isolated environment.
+ """
+ import os
+
+ root = find_project_root()
+ project_root = str(root) if (root / "pyproject.toml").exists() else ""
+
+ env: dict[str, str] = {
+ k: v
+ for k, v in os.environ.items()
+ if k not in _SYSTEM_ENV_BLOCKLIST
+ and not any(k.startswith(p) for p in _PREFIX_BLOCKLIST)
+ and not v.startswith(project_root + "/src/")
+ }
+ return env
+
+
+def inject_exgentic_env(env: dict[str, str]) -> None:
+ """Add exgentic context vars and resolved settings paths into *env*.
+
+ Mutates *env* in-place.
+ """
+ from ...core.context import context_env
+ from ...environment.instance import get_manager
+ from ...utils.settings import get_settings
+
+ for k, v in context_env().items():
+ env[k] = v
+ for key in ("EXGENTIC_CTX_OUTPUT_DIR", "EXGENTIC_CTX_CACHE_DIR"):
+ if key in env:
+ env[key] = str(Path(env[key]).resolve())
+
+ settings = get_settings()
+ # Use the EnvironmentManager's base_dir (~/.exgentic/) so that
+ # EXGENTIC_CACHE_DIR points to the same location where benchmark
+ # data is actually installed. The old settings.cache_dir default
+ # (".exgentic") resolved to a CWD-relative path that diverged from
+ # the manager's absolute ~/.exgentic/ path, breaking Docker mounts.
+ manager = get_manager()
+ env.setdefault("EXGENTIC_CACHE_DIR", str(manager.base_dir))
+ env.setdefault("EXGENTIC_OUTPUT_DIR", str(Path(settings.output_dir).resolve()))
+
+
+def make_close(transport: Any, stop_fn: Any) -> Any:
+ """Create a close function for an ObjectProxy.
+
+ Attempts a graceful ``close()`` on the remote object, then shuts
+ down the transport and calls *stop_fn* to tear down the underlying
+ process/container.
+ """
+
+ def _close() -> None:
+ try:
+ transport.call("close")
+ except Exception:
+ pass
+ transport.close()
+ stop_fn()
+
+ return _close
diff --git a/labs/AgentStream/exgentic/src/exgentic/adapters/runners/direct.py b/labs/AgentStream/exgentic/src/exgentic/adapters/runners/direct.py
new file mode 100644
index 00000000..c00787ae
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/adapters/runners/direct.py
@@ -0,0 +1,35 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+"""DirectTransport — no isolation, calls the object in the same thread."""
+
+from __future__ import annotations
+
+from typing import Any
+
+from .transport import ObjectHost, Transport
+
+
+class DirectTransport(Transport):
+ """Calls the object directly in the same thread and process.
+
+ Useful as a baseline and as the default runner.
+ """
+
+ def __init__(self, obj: Any) -> None:
+ self._host = ObjectHost(obj)
+
+ def call(self, method: str, *args: Any, **kwargs: Any) -> Any:
+ return self._host.handle("call", method, *args, **kwargs)
+
+ def get(self, name: str) -> Any:
+ return self._host.handle("get", name)
+
+ def set(self, name: str, value: Any) -> None:
+ self._host.handle("set", name, value)
+
+ def close(self) -> None:
+ pass
+
+ def __repr__(self) -> str:
+ return f"DirectTransport({self._host.obj!r})"
diff --git a/labs/AgentStream/exgentic/src/exgentic/adapters/runners/docker.py b/labs/AgentStream/exgentic/src/exgentic/adapters/runners/docker.py
new file mode 100644
index 00000000..4ac7822b
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/adapters/runners/docker.py
@@ -0,0 +1,227 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+"""DockerRunner — runs the HTTP service inside a Docker container.
+
+Uses the same HTTPTransport as ServiceRunner, but the uvicorn server
+runs inside a container instead of a local thread.
+
+The Docker image is managed by the EnvironmentManager — DockerRunner
+only starts the container and wires up volumes, ports and env vars.
+"""
+
+from __future__ import annotations
+
+import atexit
+import shutil
+import subprocess
+from pathlib import Path
+from typing import Any
+
+from ._utils import (
+ find_free_port,
+ inject_exgentic_env,
+ make_close,
+ prepare_subprocess_env,
+ serialize_kwargs,
+)
+from .service import HTTPTransport, _wait_for_health
+from .transport import ObjectProxy
+
+
+def _docker(*args: str, check: bool = True, **kwargs: Any) -> subprocess.CompletedProcess:
+ docker_bin = shutil.which("docker")
+ if docker_bin is None:
+ raise RuntimeError("docker CLI not found on PATH")
+ return subprocess.run([docker_bin, *args], check=check, **kwargs)
+
+
+class DockerRunner:
+ """Start a containerised HTTP service and return an ObjectProxy.
+
+ Parameters
+ ----------
+ target_cls: Class to instantiate inside the container.
+ env_name: Environment name for EnvironmentManager (e.g. "benchmarks/bfcl").
+ module_path: Dotted module path for locating package resources.
+ image: Pre-built image name (skips EM lookup).
+ dockerfile: Path to a Dockerfile to build from.
+ port: Host port to bind (auto-selected if None).
+ docker_args: Extra arguments forwarded to ``docker run``.
+ dependencies: Pip packages to install in the image.
+ docker_socket: Mount the host Docker socket into the container.
+ volumes: Host-to-container volume mappings (``{host: container}``).
+ """
+
+ def __init__(
+ self,
+ target_cls: type | str,
+ *args: Any,
+ env_name: str = "",
+ module_path: str = "",
+ image: str | None = None,
+ dockerfile: str | None = None,
+ port: int | None = None,
+ docker_args: list[str] | None = None,
+ dependencies: list[str] | None = None,
+ docker_socket: bool = False,
+ volumes: dict[str, str] | None = None,
+ **kwargs: Any,
+ ) -> None:
+ if args:
+ raise ValueError(
+ "DockerRunner requires keyword-only constructor arguments. "
+ "Pass all arguments as kwargs instead of positional args."
+ )
+ self._target_cls = target_cls
+ self._kwargs = kwargs
+ self._env_name = env_name
+ self._module_path = module_path
+ self._image = image
+ self._dockerfile = dockerfile
+ self._port = port or find_free_port()
+ self._docker_args = docker_args or []
+ self._dependencies = dependencies or []
+ self._docker_socket = docker_socket
+ self._volumes = volumes or {}
+ self._container_id: str | None = None
+
+ # ── image handling ───────────────────────────────────────────────
+
+ def _ensure_image(self) -> str:
+ if self._image:
+ return self._image
+
+ if self._dockerfile:
+ tag = f"exgentic-runner-custom:{hash(self._dockerfile) & 0xFFFFFFFF:08x}"
+ path = Path(self._dockerfile)
+ _docker("build", "-t", tag, "-f", str(path), str(path.parent), capture_output=True)
+ return tag
+
+ if not self._env_name:
+ raise RuntimeError(
+ "DockerRunner requires 'env_name' (and usually 'module_path') "
+ "when no 'image' or 'dockerfile' is provided."
+ )
+
+ # Use EM's pre-built image.
+ from ...environment.instance import get_manager
+
+ mgr = get_manager()
+ image = mgr.docker_image(self._env_name)
+ if image:
+ return image
+
+ # Not pre-installed — install via EM now.
+ from ...environment import EnvType
+ from ...environment.helpers import get_exgentic_install_target
+
+ project_root, packages = get_exgentic_install_target()
+ all_packages = (packages or []) + list(self._dependencies)
+ mgr.install(
+ self._env_name,
+ env_type=EnvType.DOCKER,
+ module_path=self._module_path,
+ docker_socket=self._docker_socket,
+ project_root=project_root,
+ packages=all_packages or None,
+ )
+ image = mgr.docker_image(self._env_name)
+ if not image:
+ raise RuntimeError(f"EM install succeeded but no Docker image found for {self._env_name}")
+ return image
+
+ # ── container lifecycle ──────────────────────────────────────────
+
+ def start(self) -> ObjectProxy:
+ image = self._ensure_image()
+
+ if isinstance(self._target_cls, str):
+ cls_ref = self._target_cls
+ else:
+ cls_ref = f"{self._target_cls.__module__}:{self._target_cls.__qualname__}"
+ kwargs_flag, kwargs_value = serialize_kwargs(self._kwargs)
+
+ run_args: list[str] = ["run", "-d", "-p", f"{self._port}:8080"]
+
+ # Forward host environment into the container (API tokens, user
+ # config) while excluding system-level and IDE vars.
+ env = prepare_subprocess_env()
+ inject_exgentic_env(env)
+ cache_dir = env.get("EXGENTIC_CACHE_DIR", "")
+
+ for k, v in env.items():
+ run_args.extend(["-e", f"{k}={v}"])
+
+ # Mount Docker socket for sibling container access.
+ if self._docker_socket:
+ run_args.extend(["-v", "/var/run/docker.sock:/var/run/docker.sock"])
+
+ # Always mount the cache dir so benchmarks that skip data downloads
+ # during Docker build (e.g. browsecompplus) can access host-side data,
+ # and benchmarks that bake data into the image (e.g. appworld) can
+ # also work since the volume mount overlays the image path.
+ Path(cache_dir).mkdir(parents=True, exist_ok=True)
+ run_args.extend(["-v", f"{cache_dir}:{cache_dir}"])
+
+ # Mount volumes. Resolve to absolute paths (Docker requires them)
+ # and ensure source directories exist — Docker Desktop on macOS
+ # cannot create mount sources in some protected paths.
+ for host_path, container_path in self._volumes.items():
+ host_path = str(Path(host_path).resolve())
+ container_path = str(Path(container_path)) if Path(container_path).is_absolute() else container_path
+ Path(host_path).mkdir(parents=True, exist_ok=True)
+ run_args.extend(["-v", f"{host_path}:{container_path}"])
+
+ run_args.extend(self._docker_args)
+ run_args.extend(
+ [
+ image,
+ "exgentic",
+ "serve",
+ "--cls",
+ cls_ref,
+ kwargs_flag,
+ kwargs_value,
+ "--host",
+ "0.0.0.0",
+ "--port",
+ "8080",
+ ]
+ )
+
+ result = _docker(*run_args, capture_output=True, text=True)
+ self._container_id = result.stdout.strip()
+ atexit.register(self._stop_container)
+
+ url = f"http://127.0.0.1:{self._port}"
+ try:
+ _wait_for_health(url, timeout=60.0)
+ except TimeoutError:
+ cid = self._container_id or ""
+ logs = _docker("logs", cid, check=False, capture_output=True, text=True)
+ status = _docker(
+ "inspect", "--format", "{{.State.Status}}", cid, check=False, capture_output=True, text=True
+ )
+ self._stop_container()
+ raise TimeoutError(
+ f"Container did not become healthy within 60s.\n"
+ f"Status: {status.stdout.strip()}\n"
+ f"Logs:\n{logs.stdout}\n{logs.stderr}"
+ ) from None
+
+ transport = HTTPTransport(url, timeout=600.0)
+ proxy = ObjectProxy(transport)
+ object.__setattr__(proxy, "close", make_close(transport, self._stop_container))
+ return proxy
+
+ def _stop_container(self) -> None:
+ if self._container_id is None:
+ return
+ cid = self._container_id
+ self._container_id = None
+ try:
+ _docker("stop", "-t", "2", cid, check=False, capture_output=True)
+ _docker("rm", "-f", cid, check=False, capture_output=True)
+ except Exception:
+ pass
diff --git a/labs/AgentStream/exgentic/src/exgentic/adapters/runners/process.py b/labs/AgentStream/exgentic/src/exgentic/adapters/runners/process.py
new file mode 100644
index 00000000..ed9560b4
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/adapters/runners/process.py
@@ -0,0 +1,174 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+"""PipeTransport — runs the object in a subprocess via multiprocessing + cloudpickle."""
+
+from __future__ import annotations
+
+import multiprocessing as mp
+import weakref
+from typing import Any
+
+import cloudpickle as cp
+
+from .transport import ObjectHost, Transport, deserialize_error, serialize_error
+
+# ── worker process ───────────────────────────────────────────────────
+
+
+def _worker(q_in: mp.Queue, q_out: mp.Queue) -> None:
+ """Subprocess entry point: create the object and serve RPC requests."""
+ # Late imports — these run in the child process.
+ from ...core.context import init_context_from_env, set_context, try_get_context
+ from ...observers.logging import configure_warnings_logging
+
+ configure_warnings_logging(replace_existing_file_handlers=False)
+
+ try:
+ tag, target_cls, args, kwargs, ctx = cp.loads(q_in.get())
+ assert tag == "init"
+
+ # Restore context in child process.
+ if ctx is not None:
+ set_context(ctx)
+ else:
+ try:
+ init_context_from_env()
+ except RuntimeError:
+ pass # No context env vars — standalone worker
+
+ obj = target_cls(*args, **kwargs)
+
+ # If the object has a session_id, update context to include it.
+ session_id = getattr(obj, "session_id", None)
+ if session_id:
+ current_ctx = try_get_context()
+ if current_ctx is not None:
+ set_context(current_ctx.with_session(str(session_id)))
+
+ q_out.put(cp.dumps(("ready", None)))
+ except Exception as exc:
+ q_out.put(cp.dumps(("error", serialize_error(exc))))
+ return
+
+ host = ObjectHost(obj)
+ while True:
+ try:
+ raw = q_in.get()
+ if raw is None: # shutdown sentinel
+ break
+ op, name, args, kwargs = cp.loads(raw)
+ try:
+ result = host.handle(op, name, *args, **kwargs)
+ q_out.put(cp.dumps(("ok", result)))
+ except Exception as exc:
+ q_out.put(cp.dumps(("error", serialize_error(exc))))
+ except (EOFError, BrokenPipeError):
+ break
+
+
+# ── transport ────────────────────────────────────────────────────────
+
+
+class PipeTransport(Transport):
+ """Runs the target in a child process with full memory isolation.
+
+ Uses cloudpickle for serialization and multiprocessing queues
+ for communication. Propagates the exgentic Context to the child.
+ """
+
+ def __init__(self, target_cls: type, *args: Any, **kwargs: Any) -> None:
+ self._target_cls = target_cls
+ self._args = args
+ self._kwargs = kwargs
+ self._ctx = mp.get_context("spawn")
+ self._q_in: mp.Queue | None = None
+ self._q_out: mp.Queue | None = None
+ self._proc: mp.Process | None = None
+
+ def start(self) -> None:
+ if self._proc is not None and self._proc.is_alive():
+ return
+
+ from ...core.context import context_env_scope, try_get_context
+
+ self._q_in = self._ctx.Queue()
+ self._q_out = self._ctx.Queue()
+ self._proc = self._ctx.Process(
+ target=_worker,
+ args=(self._q_in, self._q_out),
+ daemon=True,
+ )
+ # Ensure context env vars are in os.environ for the spawned process.
+ with context_env_scope():
+ self._proc.start()
+ self._finalizer = weakref.finalize(self, _terminate, self._q_in, self._proc)
+
+ # Send init payload with context.
+ ctx = try_get_context()
+ self._q_in.put(cp.dumps(("init", self._target_cls, self._args, self._kwargs, ctx)))
+ status, payload = self._recv()
+ if status == "error":
+ self.close()
+ raise deserialize_error(payload)
+
+ # ── internal helpers ─────────────────────────────────────────────
+
+ def _recv(self) -> tuple[str, Any]:
+ assert self._q_out is not None
+ if self._proc is not None and not self._proc.is_alive():
+ raise RuntimeError(f"Worker process died (exit code: {self._proc.exitcode})")
+ return cp.loads(self._q_out.get())
+
+ def _rpc(self, op: str, name: str, *args: Any, **kwargs: Any) -> Any:
+ if self._proc is None or self._q_in is None or not self._proc.is_alive():
+ raise RuntimeError("Worker process is not running")
+ self._q_in.put(cp.dumps((op, name, args, kwargs)))
+ status, payload = self._recv()
+ if status == "error":
+ raise deserialize_error(payload)
+ return payload
+
+ # ── Transport API ────────────────────────────────────────────────
+
+ def call(self, method: str, *args: Any, **kwargs: Any) -> Any:
+ return self._rpc("call", method, *args, **kwargs)
+
+ def get(self, name: str) -> Any:
+ return self._rpc("get", name)
+
+ def set(self, name: str, value: Any) -> None:
+ self._rpc("set", name, value)
+
+ def close(self) -> None:
+ _terminate(self._q_in, self._proc)
+ self._q_in = None
+ self._q_out = None
+ self._proc = None
+ try:
+ self._finalizer.detach()
+ except Exception:
+ pass
+
+ def __repr__(self) -> str:
+ pid = self._proc.pid if self._proc else None
+ return f"PipeTransport({self._target_cls.__name__}, pid={pid})"
+
+
+def _terminate(q_in: mp.Queue | None, proc: mp.Process | None) -> None:
+ """Shut down the worker process (used by both close() and the weak finalizer)."""
+ try:
+ if q_in is not None:
+ q_in.put(None)
+ except Exception:
+ pass
+ try:
+ if proc is not None:
+ proc.join(timeout=2.0)
+ except Exception:
+ pass
+ try:
+ if proc is not None and proc.is_alive():
+ proc.terminate()
+ except Exception:
+ pass
diff --git a/labs/AgentStream/exgentic/src/exgentic/adapters/runners/service.py b/labs/AgentStream/exgentic/src/exgentic/adapters/runners/service.py
new file mode 100644
index 00000000..fa530755
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/adapters/runners/service.py
@@ -0,0 +1,237 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+"""HTTPTransport + serve() — run any object as an HTTP service."""
+
+from __future__ import annotations
+
+import base64
+import threading
+import time
+from typing import Any, Optional
+
+import cloudpickle as cp
+import httpx
+from fastapi import FastAPI
+from pydantic import BaseModel as PydanticBaseModel
+
+from .transport import ObjectHost, ObjectProxy, Transport, deserialize_error, serialize_error
+
+# ── HTTP models ──────────────────────────────────────────────────────
+
+
+class CallRequest(PydanticBaseModel):
+ method: str
+ args: str # base64(cloudpickle)
+ kwargs: str # base64(cloudpickle)
+
+
+class GetRequest(PydanticBaseModel):
+ name: str
+
+
+class SetRequest(PydanticBaseModel):
+ name: str
+ value: str # base64(cloudpickle)
+
+
+class RPCResponse(PydanticBaseModel):
+ status: str # "ok" | "error"
+ result: Optional[str] = None # base64(cloudpickle)
+ error_type: Optional[str] = None
+ error_msg: Optional[str] = None
+ error_tb: Optional[str] = None
+ error_pickled: Optional[str] = None # base64(cloudpickle'd exception)
+
+
+# ── helpers ──────────────────────────────────────────────────────────
+
+
+def _encode(obj: Any) -> str:
+ return base64.b64encode(cp.dumps(obj)).decode("ascii")
+
+
+def _decode(data: str) -> Any:
+ return cp.loads(base64.b64decode(data))
+
+
+def _error_response(exc: Exception) -> RPCResponse:
+ data = serialize_error(exc)
+ pickled_b64 = None
+ if data["pickled"] is not None:
+ pickled_b64 = base64.b64encode(data["pickled"]).decode("ascii")
+ return RPCResponse(
+ status="error",
+ error_type=data["type"],
+ error_msg=data["msg"],
+ error_tb=data["tb"],
+ error_pickled=pickled_b64,
+ )
+
+
+# ── FastAPI app ──────────────────────────────────────────────────────
+
+
+def create_app(host: ObjectHost) -> FastAPI:
+ app = FastAPI()
+
+ @app.get("/health")
+ def health():
+ return {"status": "ok"}
+
+ @app.post("/call")
+ def handle_call(req: CallRequest) -> RPCResponse:
+ try:
+ result = host.handle("call", req.method, *_decode(req.args), **_decode(req.kwargs))
+ return RPCResponse(status="ok", result=_encode(result))
+ except Exception as exc:
+ return _error_response(exc)
+
+ @app.post("/get")
+ def handle_get(req: GetRequest) -> RPCResponse:
+ try:
+ return RPCResponse(status="ok", result=_encode(host.handle("get", req.name)))
+ except Exception as exc:
+ return _error_response(exc)
+
+ @app.post("/set")
+ def handle_set(req: SetRequest) -> RPCResponse:
+ try:
+ host.handle("set", req.name, _decode(req.value))
+ return RPCResponse(status="ok")
+ except Exception as exc:
+ return _error_response(exc)
+
+ return app
+
+
+# ── serve() ──────────────────────────────────────────────────────────
+
+
+def serve(obj: Any, host: str = "0.0.0.0", port: int = 8080) -> None:
+ """Serve an object over HTTP (blocking)."""
+ import uvicorn
+
+ uvicorn.run(create_app(ObjectHost(obj)), host=host, port=port, log_level="warning")
+
+
+# ── HTTPTransport — client side ──────────────────────────────────────
+
+
+class HTTPTransport(Transport):
+ """Talks to an HTTP server hosting an ObjectHost."""
+
+ def __init__(self, base_url: str, timeout: float = 30.0) -> None:
+ self._base_url = base_url.rstrip("/")
+ self._client = httpx.Client(timeout=timeout)
+
+ def _rpc(self, endpoint: str, payload: dict) -> Any:
+ resp = self._client.post(f"{self._base_url}{endpoint}", json=payload)
+ resp.raise_for_status()
+ data = RPCResponse(**resp.json())
+ if data.status == "error":
+ pickled = base64.b64decode(data.error_pickled) if data.error_pickled else None
+ raise deserialize_error(
+ {
+ "type": data.error_type or "RuntimeError",
+ "msg": data.error_msg or "",
+ "tb": data.error_tb or "",
+ "pickled": pickled,
+ }
+ )
+ return _decode(data.result) if data.result is not None else None
+
+ def call(self, method: str, *args: Any, **kwargs: Any) -> Any:
+ return self._rpc(
+ "/call",
+ {
+ "method": method,
+ "args": _encode(args),
+ "kwargs": _encode(kwargs),
+ },
+ )
+
+ def get(self, name: str) -> Any:
+ return self._rpc("/get", {"name": name})
+
+ def set(self, name: str, value: Any) -> None:
+ self._rpc("/set", {"name": name, "value": _encode(value)})
+
+ def close(self) -> None:
+ self._client.close()
+
+ def __repr__(self) -> str:
+ return f"HTTPTransport({self._base_url!r})"
+
+
+# ── Utilities ────────────────────────────────────────────────────────
+
+
+def _wait_for_health(url: str, timeout: float = 15.0) -> None:
+ deadline = time.monotonic() + timeout
+ while time.monotonic() < deadline:
+ try:
+ if httpx.get(f"{url}/health", timeout=2.0).status_code == 200:
+ return
+ except httpx.HTTPError:
+ pass
+ time.sleep(0.1)
+ raise TimeoutError(f"Service at {url} did not become healthy within {timeout}s")
+
+
+# ── ServiceRunner ────────────────────────────────────────────────────
+
+
+class ServiceRunner:
+ """Starts an HTTP service in a background thread and returns an ObjectProxy."""
+
+ def __init__(
+ self,
+ target_cls: type,
+ *args: Any,
+ port: int | None = None,
+ **kwargs: Any,
+ ) -> None:
+ self._target_cls = target_cls
+ self._args = args
+ self._kwargs = kwargs
+ from ._utils import find_free_port
+
+ self._port = port or find_free_port()
+ self._server = None
+
+ def start(self) -> ObjectProxy:
+ import uvicorn
+
+ from ...core.context import set_context_fallback, try_get_context
+
+ # Set process-wide fallback so context is available in uvicorn's
+ # request handler threads (which don't inherit ContextVar).
+ set_context_fallback(try_get_context())
+
+ obj = self._target_cls(*self._args, **self._kwargs)
+ app = create_app(ObjectHost(obj))
+
+ config = uvicorn.Config(app, host="127.0.0.1", port=self._port, log_level="warning")
+ self._server = uvicorn.Server(config)
+ threading.Thread(target=self._server.run, daemon=True).start()
+
+ url = f"http://127.0.0.1:{self._port}"
+ _wait_for_health(url)
+
+ transport = HTTPTransport(url)
+ proxy = ObjectProxy(transport)
+
+ server_ref = self._server
+
+ def _close() -> None:
+ try:
+ transport.call("close")
+ except AttributeError:
+ pass
+ transport.close()
+ server_ref.should_exit = True
+ set_context_fallback(None)
+
+ object.__setattr__(proxy, "close", _close)
+ return proxy
diff --git a/labs/AgentStream/exgentic/src/exgentic/adapters/runners/thread.py b/labs/AgentStream/exgentic/src/exgentic/adapters/runners/thread.py
new file mode 100644
index 00000000..dda6fa39
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/adapters/runners/thread.py
@@ -0,0 +1,92 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+"""ThreadTransport — runs the object in a dedicated thread, queue-based RPC."""
+
+from __future__ import annotations
+
+import contextvars
+import queue
+import threading
+from typing import Any
+
+from .transport import ObjectHost, Transport, deserialize_error, serialize_error
+
+_SHUTDOWN = object()
+
+
+class ThreadTransport(Transport):
+ """Runs the target object in a dedicated daemon thread.
+
+ Communication happens via two queues (request / response).
+ The object's methods never block the caller's thread except
+ while waiting for the result.
+ """
+
+ def __init__(self, target_cls: type, *args: Any, **kwargs: Any) -> None:
+ self._target_cls = target_cls
+ self._args = args
+ self._kwargs = kwargs
+ self._req: queue.Queue = queue.Queue()
+ self._resp: queue.Queue = queue.Queue()
+ self._thread: threading.Thread | None = None
+
+ def start(self) -> None:
+ ctx = contextvars.copy_context()
+ self._thread = threading.Thread(target=ctx.run, args=(self._worker,), daemon=True)
+ self._thread.start()
+ status, payload = self._resp.get()
+ if status == "error":
+ raise deserialize_error(payload)
+
+ # ── worker loop ──────────────────────────────────────────────────
+
+ def _worker(self) -> None:
+ try:
+ obj = self._target_cls(*self._args, **self._kwargs)
+ except Exception as exc:
+ self._resp.put(("error", serialize_error(exc)))
+ return
+
+ host = ObjectHost(obj)
+ self._resp.put(("ready", None))
+
+ while True:
+ msg = self._req.get()
+ if msg is _SHUTDOWN:
+ break
+ op, name, args, kwargs = msg
+ try:
+ result = host.handle(op, name, *args, **kwargs)
+ self._resp.put(("ok", result))
+ except Exception as exc:
+ self._resp.put(("error", serialize_error(exc)))
+
+ # ── Transport API ────────────────────────────────────────────────
+
+ def _rpc(self, op: str, name: str, *args: Any, **kwargs: Any) -> Any:
+ if self._thread is None or not self._thread.is_alive():
+ raise RuntimeError("Worker thread is not running")
+ self._req.put((op, name, args, kwargs))
+ status, payload = self._resp.get()
+ if status == "error":
+ raise deserialize_error(payload)
+ return payload
+
+ def call(self, method: str, *args: Any, **kwargs: Any) -> Any:
+ return self._rpc("call", method, *args, **kwargs)
+
+ def get(self, name: str) -> Any:
+ return self._rpc("get", name)
+
+ def set(self, name: str, value: Any) -> None:
+ self._rpc("set", name, value)
+
+ def close(self) -> None:
+ if self._thread is not None and self._thread.is_alive():
+ self._req.put(_SHUTDOWN)
+ self._thread.join(timeout=5.0)
+ self._thread = None
+
+ def __repr__(self) -> str:
+ return f"ThreadTransport({self._target_cls.__name__})"
diff --git a/labs/AgentStream/exgentic/src/exgentic/adapters/runners/transport.py b/labs/AgentStream/exgentic/src/exgentic/adapters/runners/transport.py
new file mode 100644
index 00000000..fe4887d2
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/adapters/runners/transport.py
@@ -0,0 +1,188 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+"""Core transport abstractions: Transport, ObjectHost, ObjectProxy, and error helpers."""
+
+from __future__ import annotations
+
+import builtins
+import inspect
+import traceback
+from abc import ABC, abstractmethod
+from typing import Any
+
+import cloudpickle as cp
+
+# Sentinel returned by ``get`` when the attribute is a bound method.
+# The proxy checks for this to avoid serialising the entire instance.
+CALLABLE_MARKER = {"__exgentic_callable__": True}
+
+# ── Transport interface ──────────────────────────────────────────────
+
+
+class Transport(ABC):
+ """Communication channel between a proxy and a remote object.
+
+ Every transport implements four operations so that ``ObjectProxy``
+ can forward attribute access and method calls regardless of where
+ the real object lives.
+ """
+
+ @abstractmethod
+ def call(self, method: str, *args: Any, **kwargs: Any) -> Any:
+ ...
+
+ @abstractmethod
+ def get(self, name: str) -> Any:
+ ...
+
+ @abstractmethod
+ def set(self, name: str, value: Any) -> None:
+ ...
+
+ @abstractmethod
+ def close(self) -> None:
+ ...
+
+
+# ── ObjectHost — server side ─────────────────────────────────────────
+
+
+class ObjectHost:
+ """Executes operations on a real object (the "server side").
+
+ Used identically whether the object lives in the same thread,
+ a subprocess, or an HTTP server.
+ """
+
+ def __init__(self, obj: Any) -> None:
+ self.obj = obj
+
+ def handle(self, op: str, name: str, *args: Any, **kwargs: Any) -> Any:
+ if op == "call":
+ return getattr(self.obj, name)(*args, **kwargs)
+ if op == "get":
+ value = getattr(self.obj, name)
+ # Bound methods cannot be reliably serialised (the instance may
+ # contain locks, threads, etc.). Return a lightweight marker so
+ # the proxy knows to use ``call`` instead.
+ if inspect.ismethod(value) or inspect.isbuiltin(value):
+ return CALLABLE_MARKER
+ return value
+ if op == "set":
+ setattr(self.obj, name, args[0])
+ return None
+ if op == "del":
+ delattr(self.obj, name)
+ return None
+ raise ValueError(f"Unknown operation: {op!r}")
+
+
+# ── Error serialization ──────────────────────────────────────────────
+
+
+def serialize_error(exc: BaseException) -> dict:
+ """Serialize an exception into a dict that can cross process/network boundaries.
+
+ The dict always contains string fallbacks (``type``, ``msg``, ``tb``).
+ When possible it also includes a ``pickled`` copy of the original
+ exception so that custom exception types and attributes survive.
+ """
+ pickled = None
+ try:
+ pickled = cp.dumps(exc)
+ except Exception:
+ pass
+ return {
+ "type": type(exc).__qualname__,
+ "msg": str(exc),
+ "tb": traceback.format_exc(),
+ "pickled": pickled,
+ }
+
+
+def deserialize_error(data: dict) -> BaseException:
+ """Reconstruct an exception from a ``serialize_error`` dict.
+
+ Strategy: try cloudpickle first (preserves custom types and state),
+ then fall back to reconstructing a builtin type from its name.
+ A ``__remote_traceback__`` attribute is always attached.
+ """
+ tb = data.get("tb", "")
+
+ # Fast path: unpickle the original exception.
+ pickled = data.get("pickled")
+ if pickled is not None:
+ try:
+ exc = cp.loads(pickled)
+ if isinstance(exc, BaseException):
+ exc.__remote_traceback__ = tb # type: ignore[attr-defined]
+ return exc
+ except Exception:
+ pass
+
+ # Fallback: reconstruct from type name (builtins only) + message.
+ name = data.get("type", "RuntimeError")
+ msg = data.get("msg", "")
+ cls = getattr(builtins, name, None)
+ if not (isinstance(cls, type) and issubclass(cls, BaseException)):
+ cls = RuntimeError
+ try:
+ exc = cls(msg)
+ except TypeError:
+ exc = RuntimeError(f"{name}: {msg}")
+ exc.__remote_traceback__ = tb # type: ignore[attr-defined]
+ return exc
+
+
+# ── ObjectProxy — client side ────────────────────────────────────────
+
+
+class ObjectProxy:
+ """Transparent proxy that forwards attribute access over a Transport.
+
+ Behaves like the real object: attribute reads, writes, and method
+ calls are all forwarded through the transport.
+ """
+
+ def __init__(self, transport: Transport) -> None:
+ object.__setattr__(self, "_transport", transport)
+
+ def __getattr__(self, name: str) -> Any:
+ transport: Transport = object.__getattribute__(self, "_transport")
+ value = transport.get(name)
+ if isinstance(value, dict) and value.get("__exgentic_callable__"):
+
+ def method(*args: Any, **kwargs: Any) -> Any:
+ return transport.call(name, *args, **kwargs)
+
+ method.__name__ = name # type: ignore[attr-defined]
+ return method
+ return value
+
+ def __setattr__(self, name: str, value: Any) -> None:
+ if name.startswith("_"):
+ object.__setattr__(self, name, value)
+ else:
+ transport: Transport = object.__getattribute__(self, "_transport")
+ transport.set(name, value)
+
+ def close(self) -> None:
+ """Close the remote object, then tear down the transport."""
+ transport: Transport = object.__getattribute__(self, "_transport")
+ try:
+ transport.call("close")
+ except AttributeError:
+ pass
+ transport.close()
+
+ def __del__(self) -> None:
+ try:
+ transport: Transport = object.__getattribute__(self, "_transport")
+ transport.close()
+ except Exception:
+ pass
+
+ def __repr__(self) -> str:
+ transport: Transport = object.__getattribute__(self, "_transport")
+ return f"ObjectProxy({transport!r})"
diff --git a/labs/AgentStream/exgentic/src/exgentic/adapters/runners/venv.py b/labs/AgentStream/exgentic/src/exgentic/adapters/runners/venv.py
new file mode 100644
index 00000000..28ece872
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/adapters/runners/venv.py
@@ -0,0 +1,215 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+"""VenvRunner — runs the HTTP service inside a uv virtual environment.
+
+Uses the same HTTPTransport as ServiceRunner and DockerRunner, but the
+uvicorn server runs in a subprocess with its own isolated venv instead
+of the host Python or a Docker container.
+
+The venv is created and managed by the EnvironmentManager — VenvRunner
+only starts the subprocess and optionally installs extra runtime
+dependencies.
+"""
+
+from __future__ import annotations
+
+import atexit
+import os
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+from typing import Any
+
+from ._utils import (
+ find_free_port,
+ inject_exgentic_env,
+ make_close,
+ prepare_subprocess_env,
+ serialize_kwargs,
+)
+from .service import HTTPTransport, _wait_for_health
+from .transport import ObjectProxy
+
+_HEALTH_TIMEOUT = 30.0
+_TRANSPORT_TIMEOUT = 1800.0
+
+
+def _uv(*args: str, check: bool = True, **kwargs: Any) -> subprocess.CompletedProcess:
+ uv_bin = shutil.which("uv")
+ if uv_bin is None:
+ raise RuntimeError("uv CLI not found on PATH")
+ result = subprocess.run([uv_bin, *args], check=False, **kwargs)
+ if check and result.returncode != 0:
+ stderr = getattr(result, "stderr", "") or ""
+ stdout = getattr(result, "stdout", "") or ""
+ raise RuntimeError(f"uv {' '.join(args[:3])} failed (exit {result.returncode}):\n{stderr}\n{stdout}")
+ return result
+
+
+class VenvRunner:
+ """Start an HTTP service in an isolated uv venv and return an ObjectProxy.
+
+ Parameters
+ ----------
+ target_cls: Class to instantiate inside the venv subprocess.
+ env_name: Environment name for EnvironmentManager (e.g. "benchmarks/bfcl").
+ module_path: Dotted module path for locating package resources.
+ port: Host port to bind (auto-selected if None).
+ dependencies: Extra pip packages to install in the venv at runtime.
+ health_timeout: Seconds to wait for the health endpoint.
+ """
+
+ def __init__(
+ self,
+ target_cls: type | str,
+ *args: Any,
+ env_name: str = "",
+ module_path: str = "",
+ port: int | None = None,
+ dependencies: list[str] | None = None,
+ health_timeout: float | None = None,
+ **kwargs: Any,
+ ) -> None:
+ if args:
+ raise ValueError(
+ "VenvRunner requires keyword-only constructor arguments. "
+ "Pass all arguments as kwargs instead of positional args."
+ )
+ self._target_cls = target_cls
+ self._kwargs = kwargs
+ self._env_name = env_name
+ self._module_path = module_path
+ self._port = port or find_free_port()
+ self._dependencies = dependencies or []
+ self._health_timeout = health_timeout or _HEALTH_TIMEOUT
+ self._process: subprocess.Popen | None = None
+
+ # ── venv handling ─────────────────────────────────────────────────
+
+ def _get_venv_dir(self) -> Path:
+ """Return the venv directory managed by EnvironmentManager."""
+ from ...environment.instance import get_manager
+
+ return get_manager().env_path(self._env_name) / "venv"
+
+ def _venv_python(self) -> Path:
+ """Return the path to the Python binary inside the venv."""
+ venv = self._get_venv_dir()
+ if sys.platform == "win32":
+ return venv / "Scripts" / "python.exe"
+ return venv / "bin" / "python"
+
+ def _ensure_venv(self) -> Path:
+ """Ensure the venv exists via EnvironmentManager."""
+ from ...environment import EnvType
+ from ...environment.helpers import get_exgentic_install_target
+ from ...environment.instance import get_manager
+
+ mgr = get_manager()
+ project_root, packages = get_exgentic_install_target()
+ mgr.install(
+ self._env_name,
+ env_type=EnvType.VENV,
+ module_path=self._module_path,
+ project_root=project_root,
+ packages=packages,
+ )
+ return self._get_venv_dir()
+
+ def _install_deps(self) -> None:
+ """Install extra runtime dependencies into the venv."""
+ if not self._dependencies:
+ return
+ python = self._venv_python()
+ _uv(
+ "pip",
+ "install",
+ "--python",
+ str(python),
+ "--no-cache",
+ *self._dependencies,
+ capture_output=True,
+ text=True,
+ )
+
+ # ── subprocess lifecycle ──────────────────────────────────────────
+
+ def start(self) -> ObjectProxy:
+ venv = self._ensure_venv()
+ self._install_deps()
+
+ if isinstance(self._target_cls, str):
+ cls_ref = self._target_cls
+ else:
+ cls_ref = f"{self._target_cls.__module__}:{self._target_cls.__qualname__}"
+ kwargs_flag, kwargs_value = serialize_kwargs(self._kwargs)
+
+ # Build a filtered environment (same filtering as DockerRunner).
+ env = prepare_subprocess_env()
+ env["VIRTUAL_ENV"] = str(venv)
+ venv_bin = str(venv / "bin")
+ # Prepend venv bin to the *system* PATH so external tools (docker,
+ # podman, git, …) remain reachable from within the venv subprocess.
+ system_path = os.environ.get("PATH", "")
+ env["PATH"] = venv_bin + os.pathsep + system_path
+ inject_exgentic_env(env)
+
+ exgentic_bin = self._get_venv_dir() / "bin" / "exgentic"
+ cmd = [
+ str(exgentic_bin),
+ "serve",
+ "--cls",
+ cls_ref,
+ kwargs_flag,
+ kwargs_value,
+ "--host",
+ "127.0.0.1",
+ "--port",
+ str(self._port),
+ ]
+
+ self._process = subprocess.Popen(
+ cmd,
+ env=env,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ )
+ atexit.register(self._stop_process)
+
+ url = f"http://127.0.0.1:{self._port}"
+ try:
+ _wait_for_health(url, timeout=self._health_timeout)
+ except TimeoutError:
+ proc = self._process
+ if proc is not None:
+ proc.terminate()
+ stdout, stderr = proc.communicate(timeout=5)
+ else:
+ stdout, stderr = b"", b""
+ self._stop_process()
+ raise TimeoutError(
+ f"Venv service did not become healthy within {self._health_timeout}s.\n"
+ f"stdout:\n{stdout.decode(errors='replace')}\n"
+ f"stderr:\n{stderr.decode(errors='replace')}"
+ ) from None
+
+ transport = HTTPTransport(url, timeout=_TRANSPORT_TIMEOUT)
+ proxy = ObjectProxy(transport)
+ object.__setattr__(proxy, "close", make_close(transport, self._stop_process))
+ return proxy
+
+ def _stop_process(self) -> None:
+ if self._process is None:
+ return
+ proc = self._process
+ self._process = None
+ try:
+ proc.terminate()
+ proc.wait(timeout=5)
+ except Exception:
+ try:
+ proc.kill()
+ except Exception:
+ pass
diff --git a/labs/AgentStream/exgentic/src/exgentic/adapters/schemas/__init__.py b/labs/AgentStream/exgentic/src/exgentic/adapters/schemas/__init__.py
new file mode 100644
index 00000000..ed18cc54
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/adapters/schemas/__init__.py
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+__all__ = []
diff --git a/labs/AgentStream/exgentic/src/exgentic/adapters/schemas/json_schema.py b/labs/AgentStream/exgentic/src/exgentic/adapters/schemas/json_schema.py
new file mode 100644
index 00000000..d2d6a7be
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/adapters/schemas/json_schema.py
@@ -0,0 +1,112 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from __future__ import annotations
+
+from typing import Any, Literal
+
+from json_schema_to_pydantic import create_model as _schema_to_model
+from pydantic import BaseModel
+
+
+def make_args_model_from_json_schema(name: str, parameters: dict[str, Any]) -> type[BaseModel]:
+ """Build a Pydantic v2 model from JSON Schema and verify core semantics match.
+
+ Verifies (type/required/properties), ignoring cosmetic keys like 'title'.
+ """
+ # 1) build
+ model = _schema_to_model(
+ schema=parameters,
+ base_model_type=BaseModel,
+ root_schema=None,
+ allow_undefined_array_items=False,
+ allow_undefined_type=False,
+ )
+
+ return model
+
+
+def _schema_to_type(schema: dict[str, Any]) -> Any:
+ """Best-effort map a JSON Schema fragment to a Python type annotation."""
+ if not isinstance(schema, dict):
+ return Any
+ if isinstance(schema.get("enum"), list) and schema["enum"]:
+ return _enum_type(schema["enum"]) # type: ignore[arg-type]
+ t = schema.get("type")
+ items = schema.get("items") if isinstance(schema.get("items"), dict) else None
+ return _json_type_to_py(t, items)
+
+
+def _json_type_to_py(t: Any, item_schema: dict[str, Any] | None = None):
+ """Map a JSON Schema "type" to a Python type annotation.
+
+ Supports primitives and simple containers. For arrays/objects, uses generic
+ fallbacks unless an item schema is provided for arrays.
+ """
+ from typing import Any as TAny
+ from typing import Dict as TDict
+ from typing import List as TList
+
+ if t == "string":
+ return str
+ if t == "integer":
+ return int
+ if t == "number":
+ return float
+ if t == "boolean":
+ return bool
+ if t == "array":
+ # Try to infer item type if provided, otherwise default to list[Any]
+ if isinstance(item_schema, dict):
+ inner = _schema_to_type(item_schema)
+ return TList[inner] # type: ignore[index]
+ return TList[TAny] # type: ignore[index]
+ if t == "object":
+ return TDict[str, TAny] # type: ignore[index]
+ return Any
+
+
+def _enum_type(values: list[Any]):
+ """Create a Literal type from enum values when feasible; otherwise Any."""
+ try:
+ return Literal[tuple(values)] # type: ignore[misc]
+ except TypeError:
+ # Fallback if values contain unhashables or mixed unsupported types
+ return Any
+
+
+# def make_args_model_from_param_list(name: str, params: List[Dict[str, Any]]) -> type[BaseModel]:
+# """Create a Pydantic model from AppWorld 'standard' parameter list entries."""
+# fields: Dict[str, Tuple[Any, Any]] = {}
+
+# for p in params or []:
+# pname = p["name"]
+# ptype = p.get("type")
+# required = bool(p.get("required", False))
+# default = p.get("default", ... if required else None)
+# enum_vals = p.get("enum")
+# field_kwargs: Dict[str, Any] = {}
+
+# if isinstance(enum_vals, list) and enum_vals:
+# py_t = _enum_type(enum_vals)
+# else:
+# py_t = _json_type_to_py(ptype)
+
+# if ptype in ("number", "integer"):
+# if "minimum" in p:
+# field_kwargs["ge"] = p["minimum"]
+# if "maximum" in p:
+# field_kwargs["le"] = p["maximum"]
+# if ptype == "string":
+# if "minLength" in p:
+# field_kwargs["min_length"] = p["minLength"]
+# if "maxLength" in p:
+# field_kwargs["max_length"] = p["maxLength"]
+
+# if field_kwargs:
+# annotated = Annotated[py_t, Field(**field_kwargs)] # type: ignore[misc]
+# fields[pname] = (annotated, default)
+# else:
+# fields[pname] = (py_t, default)
+
+# return create_model(f"{name}_Args", **fields)
diff --git a/labs/AgentStream/exgentic/src/exgentic/adapters/schemas/openai.py b/labs/AgentStream/exgentic/src/exgentic/adapters/schemas/openai.py
new file mode 100644
index 00000000..2292e1ab
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/adapters/schemas/openai.py
@@ -0,0 +1,67 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from __future__ import annotations
+
+from typing import Any, Literal
+
+from pydantic import create_model
+
+from ...core.types import ActionType, SingleAction
+from .json_schema import make_args_model_from_json_schema
+
+
+def openai_tools_to_action_types(tools: list[dict[str, Any]]) -> list[ActionType]:
+ """Translate OpenAI-style tools into ActionType definitions.
+
+ Builds concrete argument models from each tool's parameter schema so no
+ information is lost when emitting tools back to the LLM via ActionType.
+ """
+ actions: list[ActionType] = []
+ for t in tools:
+ if not isinstance(t, dict) or t.get("type") != "function":
+ continue
+ fn = t.get("function") or {}
+ name = fn.get("name")
+ if not isinstance(name, str):
+ continue
+ desc = fn.get("description") or ""
+ params = fn.get("parameters") or {}
+
+ args_model = make_args_model_from_json_schema(name, params)
+
+ action_model = create_model(
+ f"{name}_Action",
+ __base__=SingleAction,
+ name=(Literal[name], name),
+ arguments=(args_model, ...),
+ )
+ actions.append(ActionType(name=name, description=str(desc), cls=action_model))
+
+ if not actions:
+ raise ValueError("No OpenAI function tools provided to translate into ActionTypes")
+ return actions
+
+
+def mcp_to_openai_tool(mcp_tool: Any) -> dict[str, Any]:
+ """Converts a tool definition from a 'mcp' format into OpenAI tool schema."""
+ function_name = mcp_tool.name
+ description = mcp_tool.description or ""
+ parameters_schema = mcp_tool.inputSchema or {"type": "object", "properties": {}}
+
+ if not function_name:
+ raise ValueError("MCP tool definition is missing a 'name'.")
+
+ tool_schema = {
+ "type": "function",
+ "function": {
+ "name": function_name,
+ "description": description,
+ "parameters": parameters_schema,
+ },
+ }
+ return tool_schema
+
+
+def mcp_tools_to_openai_tools(mcp_tools: list[dict[str, Any]]) -> list[dict[str, Any]]:
+ return [mcp_to_openai_tool(mcp_tool) for mcp_tool in mcp_tools]
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/__init__.py b/labs/AgentStream/exgentic/src/exgentic/agents/__init__.py
new file mode 100644
index 00000000..d3ec05ef
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/__init__.py
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+# Bundled agents
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/a_mem/__init__.py b/labs/AgentStream/exgentic/src/exgentic/agents/a_mem/__init__.py
new file mode 100644
index 00000000..f787a20c
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/a_mem/__init__.py
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from .a_mem_agent import AMemAgent
+
+__all__ = ["AMemAgent"]
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/a_mem/a_mem_agent.py b/labs/AgentStream/exgentic/src/exgentic/agents/a_mem/a_mem_agent.py
new file mode 100644
index 00000000..398a108d
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/a_mem/a_mem_agent.py
@@ -0,0 +1,71 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+from typing import Any, ClassVar, Optional
+
+from pydantic import ConfigDict
+
+from ...core.agent import Agent
+from ...core.types import ModelSettings
+from ...utils.settings import RunnerName
+
+
+class AMemAgent(Agent):
+
+ display_name: ClassVar[str] = "A-Mem Agent"
+ slug_name: ClassVar[str] = "a_mem"
+
+ model_config = ConfigDict(arbitrary_types_allowed=True)
+
+ model: str = "gpt-4o"
+ memory_model: Optional[str] = None
+
+ retrieve_k: int = 10
+ evo_threshold: int = 100
+ embedding_model: str = "all-MiniLM-L6-v2"
+
+ shuffle_mode: str = "isolated"
+
+ benchmark_id: Optional[str] = None
+
+ enable_tool_shortlisting: bool = False
+ max_selected_tools: int = 30
+ runner: RunnerName | None = None
+ model_settings: ModelSettings | None = None
+
+ @classmethod
+ def _get_instance_class(cls):
+ from .a_mem_instance import AMemAgentInstance
+ return AMemAgentInstance
+
+ @classmethod
+ def _get_instance_class_ref(cls) -> str:
+ return "exgentic.agents.a_mem.a_mem_instance:AMemAgentInstance"
+
+ def _get_instance_kwargs(self, session_id: str) -> dict[str, Any]:
+ return {
+ "session_id": session_id,
+ "model": self.model,
+ "memory_model": self.memory_model or self.model,
+ "retrieve_k": self.retrieve_k,
+ "evo_threshold": self.evo_threshold,
+ "embedding_model": self.embedding_model,
+ "shuffle_mode": self.shuffle_mode,
+ "model_settings": self.model_settings,
+ "benchmark_id": self.benchmark_id,
+ "enable_tool_shortlisting": self.enable_tool_shortlisting,
+ "max_selected_tools": self.max_selected_tools,
+ }
+
+ @property
+ def model_name(self) -> str:
+ return str(self.model).split("/")[-1]
+
+ def get_models_names(self) -> list[str]:
+ names = [str(self.model)]
+ mm = self.memory_model or self.model
+ if mm != self.model:
+ names.append(str(mm))
+ return names
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/a_mem/a_mem_instance.py b/labs/AgentStream/exgentic/src/exgentic/agents/a_mem/a_mem_instance.py
new file mode 100644
index 00000000..4fba1fe1
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/a_mem/a_mem_instance.py
@@ -0,0 +1,726 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import json
+import re
+import time
+from datetime import datetime
+from typing import Any, Dict, List, Optional, Union
+
+import litellm
+from litellm import (
+ ChatCompletionAssistantMessage,
+ ChatCompletionSystemMessage,
+ ChatCompletionToolMessage,
+ ChatCompletionUserMessage,
+)
+
+from ...core.agent_instance import AgentInstance
+from ...core.types import (
+ Action,
+ ActionType,
+ Message,
+ MessageAction,
+ MessageObservation,
+ MessagePayload,
+ ModelSettings,
+ Observation,
+)
+from ...utils.cost import LiteLLMCostReport
+
+from .memory_note import MemoryNote
+from .memory_store import MemoryStore
+from .prompts import GENERATE_QUERY_PROMPT, parse_keywords_response
+from ..tool_shortlisting import shortlist_tools
+
+try:
+ from ...agents.litellm_tool_calling.utils import ToolCall, ToolsActionsRegistry
+except ImportError:
+ ToolsActionsRegistry = None
+ ToolCall = dict
+
+
+class AMemAgentInstance(AgentInstance):
+
+ def __init__(
+ self,
+ session_id: str,
+ model: str = "gpt-4o",
+ memory_model: str = "gpt-4o",
+ retrieve_k: int = 10,
+ evo_threshold: int = 100,
+ embedding_model: str = "all-MiniLM-L6-v2",
+ shuffle_mode: str = "isolated",
+ model_settings: Optional[ModelSettings] = None,
+ benchmark_id: Optional[str] = None,
+ enable_tool_shortlisting: bool = False,
+ max_selected_tools: int = 30,
+ ) -> None:
+ super().__init__(session_id)
+
+ self.model = model
+ self.memory_model = memory_model
+ self.retrieve_k = retrieve_k
+ self.evo_threshold = evo_threshold
+ self.embedding_model = embedding_model
+ self.shuffle_mode = shuffle_mode
+ self.benchmark_id = benchmark_id
+ self.enable_tool_shortlisting = enable_tool_shortlisting
+ self.max_selected_tools = max_selected_tools
+
+ if model_settings is None:
+ self._model_settings = ModelSettings()
+ elif isinstance(model_settings, ModelSettings):
+ self._model_settings = model_settings
+ else:
+ self._model_settings = ModelSettings()
+
+ self._cost = LiteLLMCostReport.initialize_empty(model_name=self.model)
+ self._store: Optional[MemoryStore] = None
+
+ self.messages: list[
+ Union[
+ ChatCompletionAssistantMessage,
+ ChatCompletionToolMessage,
+ ChatCompletionUserMessage,
+ ChatCompletionSystemMessage,
+ ]
+ ] = []
+ self._step_count: int = 0
+
+ self._registry: Optional[ToolsActionsRegistry] = None
+ self._all_actions: list[ActionType] = []
+
+ self._interaction_log: List[str] = []
+
+ # Memory tracking for this session
+ self._memories_added: int = 0
+ self._evolutions_triggered: int = 0
+
+ def _log_failure(
+ self, component: str, error: Exception, context: Dict[str, Any]
+ ) -> None:
+ try:
+ log_path = self.paths.agent_dir / "amem_failures.jsonl"
+ log_path.parent.mkdir(parents=True, exist_ok=True)
+ entry = {
+ "timestamp": datetime.now().isoformat(),
+ "session_id": self.session_id,
+ "component": component,
+ "error_type": type(error).__name__,
+ "error_message": str(error)[:2000],
+ **{
+ k: str(v)[:2000] if isinstance(v, str) else v
+ for k, v in context.items()
+ },
+ }
+ with open(log_path, "a", encoding="utf-8") as fh:
+ fh.write(json.dumps(entry, ensure_ascii=False) + "\n")
+ except Exception:
+ pass
+
+ def start(
+ self,
+ task: str,
+ context: Dict[str, Any],
+ actions: list[ActionType],
+ ) -> None:
+ super().start(task, context, actions)
+
+ self._all_actions = list(self.actions)
+ if ToolsActionsRegistry is not None:
+ self._registry = ToolsActionsRegistry(self._all_actions)
+
+ task_group = str(
+ context.get("task_group")
+ or context.get("task_id")
+ or context.get("task_name")
+ or "default"
+ )
+ self._store = MemoryStore.get_or_create(
+ shuffle_mode=self.shuffle_mode,
+ task_group=task_group,
+ benchmark_id=self.benchmark_id,
+ embedding_model=self.embedding_model,
+ evo_threshold=self.evo_threshold,
+ )
+ self._store.increment_session()
+
+ system_content = self._build_system_prompt_with_memories()
+ self._add_message(
+ ChatCompletionSystemMessage(role="system", content=system_content)
+ )
+
+ content_parts: list[Any] = []
+ ctx_str = ""
+ if self.context:
+ for k, v in self.context.items():
+ if isinstance(v, dict) and v.get("type") == "image_url":
+ content_parts.append({"type": "image_url", "image_url": {"url": v["data"], "detail": "high"}})
+ else:
+ ctx_str += f"\n<{k}>\n{v}\n"
+
+ text_content = f"{self.task}\n{ctx_str}"
+ if content_parts:
+ content_parts.insert(0, {"type": "text", "text": text_content})
+ self._add_message(ChatCompletionUserMessage(role="user", content=content_parts))
+ else:
+ self._add_message(
+ ChatCompletionUserMessage(role="user", content=text_content)
+ )
+
+ self._interaction_log.append(f"Task: {self.task}")
+
+ self.logger.info(
+ "A-Mem instance started store=%s session_count=%d "
+ "memory_count=%d benchmark=%s tools=%d",
+ self._store.store_id,
+ self._store.session_count,
+ self._store.memory_count,
+ self.benchmark_id or "(none)",
+ len(self._all_actions),
+ )
+
+ def react(self, observation: Optional[Observation]) -> Optional[Action]:
+ self._step_count += 1
+
+ observation_text = self._observe(observation)
+ if observation_text:
+ self._interaction_log.append(
+ f"Environment: {observation_text}"
+ )
+
+ tools = self._assistant_tools()
+ response = self._completion(
+ model=self.model,
+ messages=self.messages,
+ tools=tools if tools else None,
+ )
+
+ if response is None:
+ self.logger.error("A-Mem: LLM returned None response")
+ return None
+
+ if response.usage:
+ self._cost.update_cost_from_tokens(
+ response.usage.prompt_tokens,
+ response.usage.completion_tokens,
+ )
+
+ choice = response["choices"][0]
+ message = choice["message"]
+ finish_reason = choice.get("finish_reason")
+
+ if finish_reason == "tool_calls" and self._registry is not None:
+ tool_calls = self._extract_tool_calls(message)
+ self._add_message(
+ ChatCompletionAssistantMessage(
+ role="assistant",
+ tool_calls=[
+ {
+ "id": tc["id"],
+ "type": "function",
+ "function": {
+ "name": tc["name"],
+ "arguments": tc["arguments"],
+ },
+ }
+ for tc in tool_calls
+ ],
+ )
+ )
+ actions = self._registry.tool_calls_to_action(tool_calls)
+
+ for tc in tool_calls:
+ self._interaction_log.append(
+ f"Agent action: {tc['name']}({tc['arguments']})"
+ )
+
+ self.logger.info(
+ "A-Mem step %d: tool_calls=%s",
+ self._step_count,
+ [tc["name"] for tc in tool_calls],
+ )
+ return actions
+ else:
+ content = message.content if message.content else ""
+
+ if not content:
+ self.logger.warning(
+ "A-Mem step %d: empty content response (finish_reason=%s), "
+ "treating as agent inability to continue",
+ self._step_count, finish_reason,
+ )
+ return None
+
+ self._add_message(
+ ChatCompletionAssistantMessage(
+ role="assistant", content=content
+ )
+ )
+
+ self._interaction_log.append(
+ f"Agent says: {content}"
+ )
+
+ self.logger.info("A-Mem step %d: message response", self._step_count)
+ return MessageAction(arguments=Message(content=content))
+
+ def close(self) -> None:
+
+ store = self._store
+ if store is None:
+ return
+
+ if self._interaction_log:
+ self._store_session_memories()
+
+ store.record_learning(
+ session_id=self.session_id,
+ task_id=str(
+ self.context.get("task_id", "") if self.context else ""
+ ),
+ memories_added=self._memories_added,
+ evolutions_triggered=self._evolutions_triggered,
+ summary=(
+ f"steps={self._step_count} "
+ f"interactions={len(self._interaction_log)} "
+ f"memories_added={self._memories_added} "
+ f"evolutions={self._evolutions_triggered}"
+ ),
+ benchmark_id=self.benchmark_id or "",
+ )
+
+ try:
+ cp = str(self.paths.agent_dir / "memory_checkpoint.json")
+ store.save_checkpoint(cp)
+ mt = str(self.paths.agent_dir / "memories.txt")
+ store.save_memories_text(mt)
+ except Exception as exc:
+ self.logger.warning("A-Mem: failed to save checkpoint: %s", exc)
+
+ self.logger.info(
+ "A-Mem session closed: %d memories added, %d evolutions",
+ self._memories_added,
+ self._evolutions_triggered,
+ )
+
+ def get_cost(self) -> LiteLLMCostReport:
+ return self._cost
+
+ def _build_system_prompt_with_memories(self) -> str:
+
+ parts: List[str] = [
+ "You are an expert agent that completes tasks using available tools.",
+ "Think step-by-step before acting.",
+ "Use available tools to interact with the environment.",
+ "When you are confident in your solution, use the finish/submit tool.",
+ ]
+
+ store = self._store
+ if store is not None and store.memory_count > 0 and self.task:
+ query = self._generate_query_keywords(self.task)
+ memory_context = self._retrieve_memory_context(query)
+ if memory_context:
+ parts.append("")
+ parts.append(
+ "Based on the context below, complete the task. "
+ "Use the context to inform your decisions."
+ )
+ parts.append("")
+ parts.append(f"Context:\n{memory_context}")
+
+ return "\n".join(parts)
+
+ def _generate_query_keywords(self, question: str) -> str:
+
+ try:
+ prompt = GENERATE_QUERY_PROMPT.format(question=question)
+ response = self._memory_llm_call(prompt)
+ keywords = parse_keywords_response(response)
+ if keywords:
+ self.logger.info(
+ "A-Mem: generated query keywords: %s", keywords
+ )
+ return keywords
+ except Exception as exc:
+ self.logger.warning(
+ "A-Mem: keyword extraction failed, using raw task: %s", exc
+ )
+ return question
+
+ def _retrieve_memory_context(self, query: str) -> str:
+
+ store = self._store
+ if store is None:
+ return ""
+
+ retrieved = store.find_related_with_neighbors(query, k=self.retrieve_k)
+ if not retrieved:
+ return ""
+
+
+ model_lower = self.model.lower() if self.model else ""
+ needs_budget = "gemini" in model_lower
+
+ if needs_budget:
+ budget = 30000
+ used = 0
+ lines: List[str] = []
+ for mem in retrieved:
+ content = mem.content
+ if len(content) > 5000:
+ content = content[:5000] + "... [truncated]"
+ entry = (
+ f"memory content: {content} "
+ f"memory context: {mem.context} "
+ f"memory keywords: {mem.keywords} "
+ f"memory tags: {mem.tags}"
+ )
+ if used + len(entry) > budget and lines:
+ break
+ lines.append(entry)
+ used += len(entry)
+ self.logger.info(
+ "A-Mem: injected %d/%d retrieved memories (%d chars, budget=%d)",
+ len(lines), len(retrieved), used, budget,
+ )
+ else:
+ lines = []
+ for mem in retrieved:
+ lines.append(
+ f"memory content: {mem.content} "
+ f"memory context: {mem.context} "
+ f"memory keywords: {mem.keywords} "
+ f"memory tags: {mem.tags}"
+ )
+
+ return "\n".join(lines)
+
+ def _store_session_memories(self) -> None:
+ store = self._store
+ if store is None:
+ return
+
+ session_content = "\n".join(self._interaction_log)
+
+ if len(session_content.strip()) < 10:
+ return
+
+ try:
+ note = MemoryNote.create_with_analysis(
+ content=session_content,
+ llm_call=self._memory_llm_call,
+ )
+
+ evolved = store.add_memory(
+ note=note,
+ llm_call=self._memory_llm_call,
+ )
+
+ self._memories_added += 1
+ if evolved:
+ self._evolutions_triggered += 1
+
+ self.logger.info(
+ "A-Mem: stored session memory [%s] evolved=%s (total=%d)",
+ note.id[:8],
+ evolved,
+ store.memory_count,
+ )
+ except Exception as exc:
+ self.logger.warning(
+ "A-Mem: failed to store session memory: %s", exc
+ )
+ self._log_failure(
+ "session_memory_storage", exc, {
+ "content_preview": session_content[:500],
+ "interaction_count": len(self._interaction_log),
+ },
+ )
+
+ def _add_message(self, message: Any) -> None:
+ self.logger.debug(
+ "Adding message: role=%s", getattr(message, "role", "?")
+ )
+ self.messages.append(message)
+
+ def _observe(self, observation: Optional[Observation]) -> Optional[str]:
+
+ if observation is None:
+ return None
+
+ observations = observation.to_observation_list()
+ if observation.is_empty():
+ if not any(obs.invoking_actions for obs in observations):
+ return None
+
+ collected_texts: List[str] = []
+
+ for obs in observations:
+ if isinstance(obs, MessageObservation) and isinstance(
+ obs.result, MessagePayload
+ ):
+ self._add_message(
+ ChatCompletionUserMessage(
+ role="user", content=obs.result.message
+ )
+ )
+ collected_texts.append(obs.result.message)
+ continue
+
+ if len(obs.invoking_actions) > 0:
+ invoking = obs.invoking_actions[0]
+ if invoking.name == "message":
+ text = str(obs)
+ self._add_message(
+ ChatCompletionUserMessage(
+ role="user", content=text
+ )
+ )
+ collected_texts.append(text)
+ continue
+
+ action_id = invoking.id
+ tool_call_id = invoking.id
+ if not (
+ isinstance(tool_call_id, str)
+ and tool_call_id.startswith("call_")
+ ):
+ if self._registry is not None:
+ tool_call_id = (
+ self._registry.action_id_to_tool_call_id.get(
+ action_id, tool_call_id
+ )
+ )
+
+ value = obs.result
+ try:
+ content = json.dumps(
+ value, ensure_ascii=False, separators=(",", ":")
+ )
+ except TypeError:
+ content = str(value)
+
+ if tool_call_id is not None:
+ self._add_message(
+ ChatCompletionToolMessage(
+ role="tool",
+ tool_call_id=tool_call_id,
+ content=content,
+ )
+ )
+ else:
+ self._add_message(
+ ChatCompletionUserMessage(
+ role="user",
+ content=f"Tool result: {content}",
+ )
+ )
+ collected_texts.append(
+ f"Result of {invoking.name}: "
+ f"{self._summarize_for_memory(content)}"
+ )
+ else:
+ text = str(obs)
+ self._add_message(
+ ChatCompletionUserMessage(
+ role="user", content=text
+ )
+ )
+ collected_texts.append(text)
+
+ if collected_texts:
+ return "\n".join(collected_texts)
+ return None
+
+ def _summarize_for_memory(self, content: str) -> str:
+
+ if len(content) < 2000:
+ return content
+
+ try:
+ data = json.loads(content)
+ except (json.JSONDecodeError, ValueError):
+ if len(content) > 10000:
+ return content[:10000] + f"\n... [truncated, total {len(content)} chars]"
+ return content
+
+ if isinstance(data, str):
+ try:
+ data = json.loads(data)
+ except (json.JSONDecodeError, ValueError):
+ if len(data) > 10000:
+ return data[:10000] + f"\n... [truncated, total {len(data)} chars]"
+ return data
+
+ if isinstance(data, list) and data and isinstance(data[0], dict):
+ first = data[0]
+ snippet_key = None
+ if "snippet" in first:
+ snippet_key = "snippet"
+ elif "content" in first and "docid" in first:
+ snippet_key = "content"
+
+ if snippet_key is not None:
+ summaries: List[str] = []
+ for item in data:
+ docid = item.get("docid", "?")
+ score = item.get("score")
+ snippet = item.get(snippet_key, "")
+ title = ""
+ if isinstance(snippet, str) and snippet.startswith("---"):
+ title_match = re.search(r"title:\s*(.+)", snippet)
+ if title_match:
+ title = title_match.group(1).strip()
+ score_str = f" score:{score:.3f}" if isinstance(score, (int, float)) else ""
+ snippet_preview = snippet[:400].replace("\n", " ") if isinstance(snippet, str) else str(snippet)[:400]
+ summaries.append(
+ f"[doc:{docid}{score_str}] {title} | {snippet_preview}"
+ )
+ return "\n".join(summaries)
+
+ max_item_chars = 200
+ summaries_generic: List[str] = []
+ for i, item in enumerate(data):
+ item_str = json.dumps(item, ensure_ascii=False, separators=(",", ":"))
+ if len(item_str) > max_item_chars:
+ item_str = item_str[:max_item_chars] + "..."
+ summaries_generic.append(item_str)
+ result = f"[{len(data)} items]\n" + "\n".join(summaries_generic)
+ return result
+
+ if isinstance(data, dict) and len(content) > 5000:
+ compact = json.dumps(data, ensure_ascii=False, separators=(",", ":"))
+ if len(compact) > 5000:
+ return compact[:5000] + f"... [truncated, total {len(compact)} chars]"
+ return compact
+
+ return content
+
+ def _assistant_tools(self) -> list[dict[str, Any]]:
+ if self._registry is None:
+ return []
+ tools = self._registry.openai_tools()
+ if not self.enable_tool_shortlisting:
+ return tools
+
+ def _cost_cb(usage):
+ if usage:
+ self._cost.update_cost_from_tokens(
+ usage.prompt_tokens, usage.completion_tokens
+ )
+
+ return shortlist_tools(
+ tools=tools,
+ max_selected=self.max_selected_tools,
+ messages=self.messages,
+ completion_fn=self._completion,
+ model=self.model,
+ logger=self.logger,
+ cost_callback=_cost_cb,
+ )
+
+ @staticmethod
+ def _extract_tool_calls(message: Any) -> list[dict[str, str]]:
+ if not hasattr(message, "tool_calls") or not message.tool_calls:
+ return []
+ tool_calls = []
+ for tc in message.tool_calls:
+ tool_calls.append({
+ "name": tc.function.name,
+ "arguments": tc.function.arguments,
+ "id": tc.id,
+ })
+ return tool_calls
+
+ def _completion(self, **kwargs) -> Any:
+ call_kwargs = self._model_settings.model_dump(
+ exclude_none=True,
+ exclude={"num_retries", "retry_after", "retry_strategy"},
+ )
+ call_kwargs.update(kwargs)
+ if call_kwargs.get("tools") is None:
+ call_kwargs.pop("tools", None)
+
+ max_attempts = 3
+ for attempt in range(max_attempts):
+ try:
+ response = litellm.completion(**call_kwargs)
+ choice = response["choices"][0] if response.get("choices") else None
+ if choice:
+ msg = choice.get("message") or {}
+ has_content = bool(msg.get("content"))
+ has_tools = bool(msg.get("tool_calls"))
+ if not has_content and not has_tools:
+ if attempt + 1 < max_attempts:
+ self.logger.warning(
+ "A-Mem LLM call attempt %d/%d: empty response "
+ "(finish_reason=%s), retrying...",
+ attempt + 1, max_attempts,
+ choice.get("finish_reason"),
+ )
+ time.sleep(2 ** attempt)
+ continue
+ return response
+ except Exception as exc:
+ self.logger.warning(
+ "A-Mem LLM call attempt %d/%d failed: %s",
+ attempt + 1,
+ max_attempts,
+ exc,
+ )
+ if attempt + 1 >= max_attempts:
+ raise
+ time.sleep(2 ** attempt)
+ return None
+
+ def _llm_call_simple(
+ self,
+ model: str,
+ prompt: str,
+ ) -> str:
+ kwargs: Dict[str, Any] = {
+ "model": model,
+ "messages": [{"role": "user", "content": prompt}],
+ "temperature": 0.7,
+ "max_tokens": 1000,
+ }
+
+ max_attempts = 3
+ for attempt in range(max_attempts):
+ try:
+ resp = litellm.completion(**kwargs)
+ if resp.usage:
+ self._cost.update_cost_from_tokens(
+ resp.usage.prompt_tokens,
+ resp.usage.completion_tokens,
+ )
+ content = resp.choices[0].message.content
+ if content is None:
+ raise ValueError("LLM returned None content")
+ return content
+ except Exception as exc:
+ self.logger.warning(
+ "A-Mem simple LLM call attempt %d/%d failed: %s",
+ attempt + 1,
+ max_attempts,
+ exc,
+ )
+ if attempt + 1 >= max_attempts:
+ self._log_failure(
+ "llm_call", exc, {
+ "model": model,
+ "prompt_length": len(prompt),
+ "prompt_preview": prompt[:500],
+ "attempts": max_attempts,
+ },
+ )
+ raise
+ time.sleep(2 ** attempt)
+ return ""
+
+ def _memory_llm_call(self, prompt: str) -> str:
+ return self._llm_call_simple(self.memory_model, prompt)
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/a_mem/memory_note.py b/labs/AgentStream/exgentic/src/exgentic/agents/a_mem/memory_note.py
new file mode 100644
index 00000000..eccc5c52
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/a_mem/memory_note.py
@@ -0,0 +1,142 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import logging
+import uuid
+from dataclasses import dataclass, field
+from datetime import datetime
+from typing import Any, Callable, Dict, List, Optional
+
+from .prompts import (
+ ANALYZE_CONTENT_PROMPT,
+ FOCUSED_KEYWORDS_PROMPT,
+ heuristic_context,
+ heuristic_keywords,
+ parse_analyze_content,
+ validate_analysis_result,
+ _parse_list_items,
+)
+
+logger = logging.getLogger("amem")
+
+
+@dataclass
+class MemoryNote:
+
+ content: str
+ id: str = ""
+ keywords: List[str] = field(default_factory=list)
+ tags: List[str] = field(default_factory=list)
+ context: str = "General"
+ links: List[int] = field(default_factory=list)
+ importance_score: float = 1.0
+ retrieval_count: int = 0
+ timestamp: str = ""
+ last_accessed: str = ""
+ evolution_history: List[Dict[str, Any]] = field(default_factory=list)
+ category: str = "Uncategorized"
+
+ def __post_init__(self) -> None:
+ if not self.id:
+ self.id = str(uuid.uuid4())
+ current_time = datetime.now().strftime("%Y%m%d%H%M")
+ if not self.timestamp:
+ self.timestamp = current_time
+ if not self.last_accessed:
+ self.last_accessed = current_time
+ # Ensure context is a string
+ if isinstance(self.context, list):
+ self.context = " ".join(self.context)
+
+ @staticmethod
+ def analyze_content(
+ content: str,
+ llm_call: Callable[[str], str],
+ ) -> Dict[str, Any]:
+ prompt = ANALYZE_CONTENT_PROMPT.format(content=content)
+ try:
+ response = llm_call(prompt)
+ analysis = parse_analyze_content(response, content)
+
+ # Retry focused keywords if empty
+ if not analysis["keywords"]:
+ logger.info(
+ "Keywords empty after initial parse -- retrying with focused prompt"
+ )
+ retry_prompt = FOCUSED_KEYWORDS_PROMPT.format(content=content)
+ retry_response = llm_call(retry_prompt)
+ analysis["keywords"] = _parse_list_items(retry_response)
+
+ return validate_analysis_result(analysis, content)
+
+ except Exception as e:
+ logger.error("Error analyzing content: %s", e)
+ return {
+ "keywords": heuristic_keywords(content),
+ "context": heuristic_context(content),
+ "tags": heuristic_keywords(content, 3),
+ }
+
+ @classmethod
+ def create_with_analysis(
+ cls,
+ content: str,
+ llm_call: Callable[[str], str],
+ timestamp: Optional[str] = None,
+ importance_score: float = 1.0,
+ ) -> "MemoryNote":
+ analysis = cls.analyze_content(content, llm_call)
+ return cls(
+ content=content,
+ keywords=analysis["keywords"],
+ context=analysis["context"],
+ tags=analysis["tags"],
+ timestamp=timestamp or "",
+ importance_score=importance_score,
+ )
+
+ # ------------------------------------------------------------------
+ # Serialization
+ # ------------------------------------------------------------------
+ def to_dict(self) -> Dict[str, Any]:
+ return {
+ "id": self.id,
+ "content": self.content,
+ "keywords": self.keywords,
+ "tags": self.tags,
+ "context": self.context,
+ "links": self.links,
+ "importance_score": self.importance_score,
+ "retrieval_count": self.retrieval_count,
+ "timestamp": self.timestamp,
+ "last_accessed": self.last_accessed,
+ "evolution_history": self.evolution_history,
+ "category": self.category,
+ }
+
+ @classmethod
+ def from_dict(cls, data: Dict[str, Any]) -> "MemoryNote":
+ return cls(
+ id=data.get("id", ""),
+ content=data.get("content", ""),
+ keywords=data.get("keywords", []),
+ tags=data.get("tags", []),
+ context=data.get("context", "General"),
+ links=data.get("links", []),
+ importance_score=data.get("importance_score", 1.0),
+ retrieval_count=data.get("retrieval_count", 0),
+ timestamp=data.get("timestamp", ""),
+ last_accessed=data.get("last_accessed", ""),
+ evolution_history=data.get("evolution_history", []),
+ category=data.get("category", "Uncategorized"),
+ )
+
+ def to_retrieval_document(self) -> str:
+ return (
+ f"content:{self.content} "
+ f"context:{self.context} "
+ f"keywords: {', '.join(self.keywords)} "
+ f"tags: {', '.join(self.tags)}"
+ )
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/a_mem/memory_store.py b/labs/AgentStream/exgentic/src/exgentic/agents/a_mem/memory_store.py
new file mode 100644
index 00000000..f67e6066
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/a_mem/memory_store.py
@@ -0,0 +1,417 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import json
+import logging
+import threading
+from dataclasses import dataclass
+from datetime import datetime
+from pathlib import Path
+from typing import Any, Callable, Dict, List, Optional, Tuple
+
+from .memory_note import MemoryNote
+from .prompts import (
+ EVOLUTION_DECISION_PROMPT,
+ STRENGTHEN_DETAILS_PROMPT,
+ UPDATE_NEIGHBORS_PROMPT,
+ parse_evolution_decision,
+ parse_strengthen_details,
+ parse_update_neighbors,
+)
+from .retriever import EmbeddingRetriever
+
+logger = logging.getLogger("amem")
+
+
+@dataclass
+class LearningEvent:
+
+ session_id: str
+ task_id: str
+ step: int
+ memories_added: int
+ evolutions_triggered: int
+ summary: str
+ benchmark_id: str = ""
+
+
+class MemoryStore:
+
+ _instances: Dict[str, "MemoryStore"] = {}
+ _global_lock = threading.Lock()
+
+ @classmethod
+ def get_or_create(
+ cls,
+ shuffle_mode: str = "isolated",
+ task_group: Optional[str] = None,
+ benchmark_id: Optional[str] = None,
+ embedding_model: str = "all-MiniLM-L6-v2",
+ evo_threshold: int = 100,
+ ) -> "MemoryStore":
+ if shuffle_mode == "isolated":
+ bm = benchmark_id or task_group or "default"
+ key = f"amem_isolated_{bm}"
+ elif shuffle_mode == "sequential":
+ key = "amem_sequential_global"
+ elif shuffle_mode == "interleaved":
+ key = "amem_interleaved_global"
+ else:
+ raise ValueError(f"Unknown shuffle_mode: {shuffle_mode!r}")
+
+ with cls._global_lock:
+ if key not in cls._instances:
+ cls._instances[key] = cls(
+ store_id=key,
+ embedding_model=embedding_model,
+ evo_threshold=evo_threshold,
+ )
+ return cls._instances[key]
+
+ @classmethod
+ def reset_all(cls) -> None:
+ with cls._global_lock:
+ cls._instances.clear()
+
+ @classmethod
+ def list_stores(cls) -> Dict[str, "MemoryStore"]:
+ with cls._global_lock:
+ return dict(cls._instances)
+
+ def __init__(
+ self,
+ store_id: str,
+ embedding_model: str = "all-MiniLM-L6-v2",
+ evo_threshold: int = 100,
+ ) -> None:
+ self.store_id = store_id
+ self._lock = threading.Lock()
+
+ self._memories: Dict[str, MemoryNote] = {}
+ self._retriever = EmbeddingRetriever(embedding_model)
+ self._embedding_model = embedding_model
+ self._evo_threshold = evo_threshold
+ self._evo_cnt: int = 0
+
+ self._session_count: int = 0
+ self._history: List[LearningEvent] = []
+ self._benchmark_counts: Dict[str, int] = {}
+
+
+ @property
+ def memory_count(self) -> int:
+ with self._lock:
+ return len(self._memories)
+
+ def get_all_memories(self) -> List[MemoryNote]:
+ with self._lock:
+ return list(self._memories.values())
+
+
+ @property
+ def session_count(self) -> int:
+ with self._lock:
+ return self._session_count
+
+ def increment_session(self) -> int:
+ with self._lock:
+ self._session_count += 1
+ return self._session_count
+
+ def add_memory(
+ self,
+ note: MemoryNote,
+ llm_call: Callable[[str], str],
+ ) -> bool:
+ with self._lock:
+ evolved = self._process_and_add(note, llm_call)
+ return evolved
+
+ def find_related_with_neighbors(
+ self,
+ query: str,
+ k: int = 10,
+ ) -> List[MemoryNote]:
+ with self._lock:
+ if not self._memories:
+ return []
+ indices = self._retriever.search(query, k)
+ all_memories = list(self._memories.values())
+ seen: set[int] = set()
+ results: List[MemoryNote] = []
+
+ for i in indices:
+ if i >= len(all_memories) or i in seen:
+ continue
+ seen.add(i)
+ note = all_memories[i]
+ note.retrieval_count += 1
+ note.last_accessed = datetime.now().strftime("%Y%m%d%H%M")
+ results.append(note)
+ for link_idx in note.links:
+ if link_idx < len(all_memories) and link_idx not in seen:
+ seen.add(link_idx)
+ linked = all_memories[link_idx]
+ linked.retrieval_count += 1
+ results.append(linked)
+
+ return results
+
+ def _process_and_add(
+ self,
+ note: MemoryNote,
+ llm_call: Callable[[str], str],
+ ) -> bool:
+ neighbor_str, indices = self._find_neighbors_for_evolution(
+ note.content, k=5
+ )
+
+ evolved = False
+ if indices:
+ try:
+ evolved = self._run_evolution(note, neighbor_str, indices, llm_call)
+ except Exception as e:
+ logger.error(
+ "Evolution failed for note %s: %s -- storing without evolution",
+ note.id[:8],
+ e,
+ )
+
+ self._memories[note.id] = note
+ self._retriever.add_documents([note.to_retrieval_document()])
+
+ if evolved:
+ self._evo_cnt += 1
+ if self._evo_cnt % self._evo_threshold == 0:
+ self._consolidate()
+
+ return evolved
+
+ def _find_neighbors_for_evolution(
+ self, query: str, k: int = 5
+ ) -> Tuple[str, List[int]]:
+ if not self._memories:
+ return "", []
+
+ indices = self._retriever.search(query, k)
+ all_memories = list(self._memories.values())
+ memory_str = ""
+ for i in indices:
+ if i >= len(all_memories):
+ continue
+ m = all_memories[i]
+ memory_str += (
+ f"memory index:{i}"
+ f"\t talk start time:{m.timestamp}"
+ f"\t memory content: {m.content}"
+ f"\t memory context: {m.context}"
+ f"\t memory keywords: {m.keywords}"
+ f"\t memory tags: {m.tags}\n"
+ )
+ return memory_str, indices
+
+ def _run_evolution(
+ self,
+ note: MemoryNote,
+ neighbor_str: str,
+ indices: List[int],
+ llm_call: Callable[[str], str],
+ ) -> bool:
+ decision_prompt = EVOLUTION_DECISION_PROMPT.format(
+ context=note.context,
+ content=note.content,
+ keywords=note.keywords,
+ nearest_neighbors_memories=neighbor_str,
+ )
+ decision_response = llm_call(decision_prompt)
+ decision = parse_evolution_decision(decision_response)
+ logger.debug("Evolution decision: %s", decision)
+
+ if decision["decision"] == "NO_EVOLUTION":
+ return False
+
+ should_strengthen = decision["decision"] in (
+ "STRENGTHEN", "STRENGTHEN_AND_UPDATE"
+ )
+ should_update = decision["decision"] in (
+ "UPDATE_NEIGHBOR", "STRENGTHEN_AND_UPDATE"
+ )
+
+ if should_strengthen:
+ strengthen_prompt = STRENGTHEN_DETAILS_PROMPT.format(
+ content=note.content,
+ keywords=note.keywords,
+ nearest_neighbors_memories=neighbor_str,
+ )
+ strengthen_response = llm_call(strengthen_prompt)
+ strengthen = parse_strengthen_details(strengthen_response)
+ logger.debug("Strengthen details: %s", strengthen)
+
+ note.links.extend(strengthen["connections"])
+ if strengthen["tags"]:
+ note.tags = strengthen["tags"]
+
+ if should_update:
+ update_prompt = UPDATE_NEIGHBORS_PROMPT.format(
+ content=note.content,
+ context=note.context,
+ nearest_neighbors_memories=neighbor_str,
+ max_neighbor_idx=len(indices) - 1,
+ neighbor_count=len(indices),
+ )
+ update_response = llm_call(update_prompt)
+ neighbor_updates = parse_update_neighbors(
+ update_response, len(indices)
+ )
+ logger.debug("Neighbor updates: %s", neighbor_updates)
+
+ noteslist = list(self._memories.values())
+ notes_id = list(self._memories.keys())
+ for i in range(min(len(indices), len(neighbor_updates))):
+ upd = neighbor_updates[i]
+ memorytmp_idx = indices[i]
+ if memorytmp_idx >= len(noteslist):
+ continue
+ notetmp = noteslist[memorytmp_idx]
+ if upd["tags"]:
+ notetmp.tags = upd["tags"]
+ if upd["context"]:
+ notetmp.context = upd["context"]
+ self._memories[notes_id[memorytmp_idx]] = notetmp
+
+ return True
+
+ def _consolidate(self) -> None:
+ logger.info(
+ "Consolidating memory retriever (%d memories, %d evolutions)",
+ len(self._memories),
+ self._evo_cnt,
+ )
+ documents = [m.to_retrieval_document() for m in self._memories.values()]
+ self._retriever.reset(documents)
+
+ def record_learning(
+ self,
+ session_id: str,
+ task_id: str,
+ memories_added: int,
+ evolutions_triggered: int,
+ summary: str,
+ benchmark_id: str = "",
+ ) -> None:
+ with self._lock:
+ self._history.append(
+ LearningEvent(
+ session_id=session_id,
+ task_id=task_id,
+ step=self._session_count,
+ memories_added=memories_added,
+ evolutions_triggered=evolutions_triggered,
+ summary=summary[:500],
+ benchmark_id=benchmark_id,
+ )
+ )
+ if benchmark_id:
+ self._benchmark_counts[benchmark_id] = (
+ self._benchmark_counts.get(benchmark_id, 0) + 1
+ )
+
+ def save_checkpoint(self, path: str) -> None:
+ with self._lock:
+ payload = {
+ "store_id": self.store_id,
+ "memory_count": len(self._memories),
+ "evo_cnt": self._evo_cnt,
+ "session_count": self._session_count,
+ "history_len": len(self._history),
+ "benchmark_counts": dict(self._benchmark_counts),
+ "memories": {
+ mid: note.to_dict()
+ for mid, note in self._memories.items()
+ },
+ }
+ Path(path).parent.mkdir(parents=True, exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ json.dump(payload, fh, indent=2, ensure_ascii=False)
+
+ def load_checkpoint(self, path: str) -> None:
+ with open(path, "r", encoding="utf-8") as fh:
+ payload = json.load(fh)
+ with self._lock:
+ self._evo_cnt = payload.get("evo_cnt", 0)
+ self._session_count = payload.get("session_count", 0)
+ self._benchmark_counts = payload.get("benchmark_counts", {})
+ memories_data = payload.get("memories", {})
+ self._memories = {
+ mid: MemoryNote.from_dict(mdata)
+ for mid, mdata in memories_data.items()
+ }
+ if self._memories:
+ documents = [
+ m.to_retrieval_document() for m in self._memories.values()
+ ]
+ self._retriever.reset(documents)
+
+ def save_memories_text(self, path: str) -> None:
+ Path(path).parent.mkdir(parents=True, exist_ok=True)
+ with self._lock:
+ lines = [
+ f"# A-Mem Memory Store: {self.store_id}",
+ f"# Memories: {len(self._memories)}",
+ f"# Sessions: {self._session_count}",
+ f"# Evolutions: {self._evo_cnt}",
+ "",
+ ]
+ for i, note in enumerate(self._memories.values()):
+ lines.append(f"--- Memory {i + 1} [{note.id[:8]}] ---")
+ lines.append(f"Content: {note.content}")
+ lines.append(f"Context: {note.context}")
+ lines.append(f"Keywords: {', '.join(note.keywords)}")
+ lines.append(f"Tags: {', '.join(note.tags)}")
+ links_str = (
+ ", ".join(str(l) for l in note.links) if note.links else "none"
+ )
+ lines.append(f"Links: {links_str}")
+ lines.append(
+ f"Importance: {note.importance_score:.2f} "
+ f"Retrieved: {note.retrieval_count} times"
+ )
+ lines.append("")
+
+ with open(path, "w", encoding="utf-8") as fh:
+ fh.write("\n".join(lines))
+
+ def get_stats(self) -> Dict[str, Any]:
+ with self._lock:
+ total = len(self._memories)
+ if total == 0:
+ return {
+ "total_memories": 0,
+ "total_evolutions": self._evo_cnt,
+ "avg_links": 0.0,
+ "avg_keywords": 0.0,
+ "avg_importance": 0.0,
+ "most_retrieved": 0,
+ }
+ links_count = sum(
+ len(m.links) for m in self._memories.values()
+ )
+ kw_count = sum(
+ len(m.keywords) for m in self._memories.values()
+ )
+ imp_sum = sum(
+ m.importance_score for m in self._memories.values()
+ )
+ max_retr = max(
+ m.retrieval_count for m in self._memories.values()
+ )
+ return {
+ "total_memories": total,
+ "total_evolutions": self._evo_cnt,
+ "avg_links": links_count / total,
+ "avg_keywords": kw_count / total,
+ "avg_importance": imp_sum / total,
+ "most_retrieved": max_retr,
+ }
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/a_mem/prompts.py b/labs/AgentStream/exgentic/src/exgentic/agents/a_mem/prompts.py
new file mode 100644
index 00000000..8b640238
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/a_mem/prompts.py
@@ -0,0 +1,405 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import json
+import re
+import logging
+from typing import Any, Callable, Dict, List, Optional
+
+logger = logging.getLogger("amem")
+
+def strip_markdown_fences(text: str) -> str:
+ text = text.strip()
+ text = re.sub(r'^```(?:json)?\s*\n?', '', text, flags=re.MULTILINE)
+ text = re.sub(r'\n?\s*```$', '', text, flags=re.MULTILINE)
+ return text.strip()
+
+
+def parse_with_json_fallback(
+ response: str,
+ plain_text_parser: Callable,
+ *parser_args,
+) -> Any:
+ try:
+ cleaned = strip_markdown_fences(response)
+ result = json.loads(cleaned)
+ if isinstance(result, dict):
+ return result
+ except (json.JSONDecodeError, ValueError):
+ pass
+ return plain_text_parser(response, *parser_args)
+
+def _parse_list_items(text: str) -> List[str]:
+ if not text or not text.strip():
+ return []
+
+ lines = text.strip().splitlines()
+ items: List[str] = []
+
+ for line in lines:
+ line = line.strip()
+ if not line:
+ continue
+ line = re.sub(r'^[\-\*\u2022]\s*', '', line)
+ line = re.sub(r'^\d+[\.\)]\s*', '', line)
+ line = line.strip().strip('"').strip("'").strip()
+ if not line:
+ continue
+ if ',' in line:
+ for part in line.split(','):
+ part = part.strip().strip('"').strip("'").strip()
+ if part:
+ items.append(part)
+ else:
+ items.append(line)
+
+ return items
+
+
+def _extract_section(
+ text: str,
+ marker: str,
+ next_markers: Optional[List[str]] = None,
+) -> str:
+ pattern = re.compile(
+ rf'^\s*{re.escape(marker)}\s*:\s*(.*)$',
+ re.IGNORECASE | re.MULTILINE,
+ )
+ match = pattern.search(text)
+ if not match:
+ return ""
+
+ start = match.end()
+ first_line = match.group(1).strip()
+
+ end = len(text)
+ if next_markers:
+ for nm in next_markers:
+ nm_pattern = re.compile(
+ rf'^\s*{re.escape(nm)}\s*:', re.IGNORECASE | re.MULTILINE
+ )
+ nm_match = nm_pattern.search(text, start)
+ if nm_match and nm_match.start() < end:
+ end = nm_match.start()
+
+ rest = text[start:end].strip()
+ if first_line and rest:
+ return first_line + "\n" + rest
+ return first_line or rest
+
+
+_STOP_WORDS = frozenset({
+ 'the', 'a', 'an', 'is', 'are', 'was', 'were', 'be', 'been', 'being',
+ 'have', 'has', 'had', 'do', 'does', 'did', 'will', 'would', 'could',
+ 'should', 'may', 'might', 'shall', 'can', 'need', 'dare', 'ought',
+ 'used', 'to', 'of', 'in', 'for', 'on', 'with', 'at', 'by', 'from',
+ 'as', 'into', 'through', 'during', 'before', 'after', 'above',
+ 'below', 'between', 'out', 'off', 'over', 'under', 'again',
+ 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why',
+ 'how', 'all', 'both', 'each', 'few', 'more', 'most', 'other',
+ 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so',
+ 'than', 'too', 'very', 'just', 'because', 'but', 'and', 'or',
+ 'if', 'while', 'about', 'up', 'it', 'its', 'i', 'me', 'my',
+ 'you', 'your', 'he', 'she', 'they', 'we', 'this', 'that', 'these',
+ 'those', 'what', 'which', 'who', 'whom', 'says', 'said', 'speaker',
+})
+
+
+def heuristic_keywords(content: str, max_keywords: int = 5) -> List[str]:
+ words = re.findall(r'\b[a-zA-Z]{3,}\b', content)
+ scored = []
+ seen: set[str] = set()
+ for w in words:
+ w_lower = w.lower()
+ if w_lower in _STOP_WORDS or w_lower in seen:
+ continue
+ seen.add(w_lower)
+ score = 2 if w[0].isupper() else 1
+ scored.append((w_lower, score))
+ scored.sort(key=lambda x: -x[1])
+ return [w for w, _ in scored[:max_keywords]]
+
+
+def heuristic_context(content: str) -> str:
+ match = re.match(r'(.+?[.!?])\s', content)
+ if match:
+ return match.group(1).strip()
+ return content[:200].strip()
+
+
+ANALYZE_CONTENT_PROMPT = """\
+Analyze the following content and provide:
+1. KEYWORDS: The most important keywords (nouns, verbs, key concepts). \
+Order from most to least important. At least three keywords. \
+Do not include speaker names or time references.
+2. CONTEXT: One sentence summarizing the main topic, key points, and purpose.
+3. TAGS: Broad categories/themes for classification (domain, format, type). \
+At least three tags.
+
+Respond using EXACTLY this format (one section per header):
+
+KEYWORDS: keyword1, keyword2, keyword3, ...
+CONTEXT: A single sentence summarizing the content.
+TAGS: tag1, tag2, tag3, ...
+
+Content for analysis:
+{content}"""
+
+
+EVOLUTION_DECISION_PROMPT = """\
+You are an AI memory evolution agent. Analyze the new memory note and its \
+nearest neighbors to decide if evolution is needed.
+
+New memory:
+- Context: {context}
+- Content: {content}
+- Keywords: {keywords}
+
+Nearest neighbor memories:
+{nearest_neighbors_memories}
+
+Based on the relationships between the new memory and its neighbors, decide:
+- NO_EVOLUTION: The memory stands alone, no changes needed.
+- STRENGTHEN: The new memory should be linked to some neighbors and its tags updated.
+- UPDATE_NEIGHBOR: The neighbors' context/tags should be updated based on new understanding.
+- STRENGTHEN_AND_UPDATE: Both strengthen and update neighbors.
+
+Respond using EXACTLY this format:
+DECISION:
+REASON: """
+
+
+STRENGTHEN_DETAILS_PROMPT = """\
+Given the new memory and its neighbors, provide updated connections and tags.
+
+New memory:
+- Content: {content}
+- Keywords: {keywords}
+
+Neighbor memories:
+{nearest_neighbors_memories}
+
+Which neighbor indices should the new memory connect to? \
+What tags best describe this memory?
+
+Respond using EXACTLY this format:
+CONNECTIONS: 0, 2, 3
+TAGS: tag1, tag2, tag3, ..."""
+
+
+UPDATE_NEIGHBORS_PROMPT = """\
+Given the new memory and its neighbor memories, update each neighbor's \
+context and tags based on a holistic understanding of all these memories together.
+
+New memory:
+- Content: {content}
+- Context: {context}
+
+Neighbor memories:
+{nearest_neighbors_memories}
+
+For each neighbor (indexed 0 to {max_neighbor_idx}), provide updated context \
+and tags. If no change is needed, repeat the original values.
+
+Respond using EXACTLY this format (one block per neighbor):
+
+NEIGHBOR 0:
+CONTEXT: updated context sentence
+TAGS: tag1, tag2, tag3
+
+NEIGHBOR 1:
+CONTEXT: updated context sentence
+TAGS: tag1, tag2, tag3
+
+(continue for all {neighbor_count} neighbors)"""
+
+
+FOCUSED_KEYWORDS_PROMPT = """\
+List exactly 5 keywords that capture the main concepts of the following text. \
+Output only the keywords, comma-separated, nothing else.
+
+Text: {content}"""
+
+
+GENERATE_QUERY_PROMPT = """\
+Given the following question, generate several keywords separated by commas.
+
+Question: {question}
+
+Keywords:"""
+
+
+def parse_analyze_content(response: str, content: str = "") -> Dict[str, Any]:
+ def _section_parse(resp: str, content_text: str = "") -> Dict[str, Any]:
+ kw_text = _extract_section(resp, "KEYWORDS", ["CONTEXT", "TAGS"])
+ ctx_text = _extract_section(resp, "CONTEXT", ["TAGS", "KEYWORDS"])
+ tags_text = _extract_section(resp, "TAGS", ["KEYWORDS", "CONTEXT"])
+ return {
+ "keywords": _parse_list_items(kw_text),
+ "context": ctx_text.strip() if ctx_text.strip() else "",
+ "tags": _parse_list_items(tags_text),
+ }
+
+ result = parse_with_json_fallback(response, _section_parse, content)
+ return validate_analysis_result(result, content)
+
+
+def parse_evolution_decision(response: str) -> Dict[str, str]:
+ def _section_parse(resp: str) -> Dict[str, str]:
+ decision_text = _extract_section(resp, "DECISION", ["REASON"])
+ reason_text = _extract_section(resp, "REASON", ["DECISION"])
+
+ decision = decision_text.strip().upper().replace(" ", "_")
+ valid_decisions = {
+ "NO_EVOLUTION", "STRENGTHEN", "UPDATE_NEIGHBOR",
+ "STRENGTHEN_AND_UPDATE",
+ }
+ if decision not in valid_decisions:
+ resp_upper = resp.upper()
+ if "STRENGTHEN" in resp_upper and "UPDATE" in resp_upper:
+ decision = "STRENGTHEN_AND_UPDATE"
+ elif "STRENGTHEN" in resp_upper:
+ decision = "STRENGTHEN"
+ elif "UPDATE" in resp_upper:
+ decision = "UPDATE_NEIGHBOR"
+ else:
+ decision = "NO_EVOLUTION"
+ return {"decision": decision, "reason": reason_text.strip()}
+
+ result = parse_with_json_fallback(response, _section_parse)
+
+ if "should_evolve" in result:
+ should_evolve = result.get("should_evolve", False)
+ actions = result.get("actions", [])
+ if not should_evolve:
+ decision = "NO_EVOLUTION"
+ elif "strengthen" in actions and "update_neighbor" in actions:
+ decision = "STRENGTHEN_AND_UPDATE"
+ elif "strengthen" in actions:
+ decision = "STRENGTHEN"
+ elif "update_neighbor" in actions:
+ decision = "UPDATE_NEIGHBOR"
+ else:
+ decision = "NO_EVOLUTION"
+ result = {"decision": decision, "reason": ""}
+
+ if "decision" not in result:
+ result = {"decision": "NO_EVOLUTION", "reason": ""}
+
+ return result
+
+
+def parse_strengthen_details(response: str) -> Dict[str, Any]:
+ def _section_parse(resp: str) -> Dict[str, Any]:
+ conn_text = _extract_section(resp, "CONNECTIONS", ["TAGS"])
+ tags_text = _extract_section(resp, "TAGS", ["CONNECTIONS"])
+ connections = []
+ for item in _parse_list_items(conn_text):
+ try:
+ connections.append(int(item.strip()))
+ except (ValueError, TypeError):
+ pass
+ return {"connections": connections, "tags": _parse_list_items(tags_text)}
+
+ result = parse_with_json_fallback(response, _section_parse)
+
+ if "suggested_connections" in result and "connections" not in result:
+ result["connections"] = [
+ int(x)
+ for x in result.get("suggested_connections", [])
+ if isinstance(x, (int, float))
+ ]
+ if "tags_to_update" in result and "tags" not in result:
+ result["tags"] = result.get("tags_to_update", [])
+
+ result.setdefault("connections", [])
+ result.setdefault("tags", [])
+ return result
+
+
+def parse_update_neighbors(
+ response: str, num_neighbors: int
+) -> List[Dict[str, Any]]:
+ def _section_parse(
+ resp: str, n_neighbors: int
+ ) -> List[Dict[str, Any]]:
+ neighbors = []
+ for i in range(n_neighbors):
+ pattern = re.compile(rf'NEIGHBOR\s+{i}\s*:', re.IGNORECASE)
+ match = pattern.search(resp)
+ if not match:
+ neighbors.append({"context": "", "tags": []})
+ continue
+ next_pattern = re.compile(
+ rf'NEIGHBOR\s+{i + 1}\s*:', re.IGNORECASE
+ )
+ next_match = next_pattern.search(resp, match.end())
+ block_end = next_match.start() if next_match else len(resp)
+ block = resp[match.end():block_end]
+ ctx = _extract_section(block, "CONTEXT", ["TAGS"])
+ tags_text = _extract_section(block, "TAGS", ["CONTEXT"])
+ neighbors.append({
+ "context": ctx.strip(),
+ "tags": _parse_list_items(tags_text),
+ })
+ return neighbors
+
+ try:
+ cleaned = strip_markdown_fences(response)
+ data = json.loads(cleaned)
+ if isinstance(data, dict):
+ contexts = data.get("new_context_neighborhood", [])
+ tags_list = data.get("new_tags_neighborhood", [])
+ neighbors = []
+ for i in range(num_neighbors):
+ ctx = contexts[i] if i < len(contexts) else ""
+ tags = tags_list[i] if i < len(tags_list) else []
+ neighbors.append({"context": ctx, "tags": tags})
+ return neighbors
+ except (json.JSONDecodeError, ValueError):
+ pass
+
+ return _section_parse(response, num_neighbors)
+
+
+def validate_analysis_result(
+ result: Dict[str, Any], content: str = ""
+) -> Dict[str, Any]:
+ if not isinstance(result, dict):
+ result = {"keywords": [], "context": "", "tags": []}
+
+ keywords = result.get("keywords", [])
+ context = result.get("context", "")
+ tags = result.get("tags", [])
+
+ if isinstance(keywords, str):
+ keywords = _parse_list_items(keywords)
+ if isinstance(tags, str):
+ tags = _parse_list_items(tags)
+ if isinstance(context, list):
+ context = " ".join(context)
+
+ if not keywords and content:
+ keywords = heuristic_keywords(content)
+ if not context and content:
+ context = heuristic_context(content)
+ if not tags and keywords:
+ tags = keywords[:3]
+
+ result["keywords"] = keywords
+ result["context"] = context
+ result["tags"] = tags
+ return result
+
+
+def parse_keywords_response(response: str) -> str:
+ try:
+ cleaned = strip_markdown_fences(response)
+ data = json.loads(cleaned)
+ if isinstance(data, dict) and "keywords" in data:
+ return str(data["keywords"])
+ except (json.JSONDecodeError, ValueError):
+ pass
+ return response.strip()
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/a_mem/retriever.py b/labs/AgentStream/exgentic/src/exgentic/agents/a_mem/retriever.py
new file mode 100644
index 00000000..f1fd2efb
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/a_mem/retriever.py
@@ -0,0 +1,54 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import logging
+from typing import List, Optional
+
+import numpy as np
+from sentence_transformers import SentenceTransformer
+from sklearn.metrics.pairwise import cosine_similarity
+
+logger = logging.getLogger("amem")
+
+
+class EmbeddingRetriever:
+
+ def __init__(self, model_name: str = "all-MiniLM-L6-v2") -> None:
+ self.model = SentenceTransformer(model_name)
+ self.corpus: List[str] = []
+ self.embeddings: Optional[np.ndarray] = None
+
+ def add_documents(self, documents: List[str]) -> None:
+ if not documents:
+ return
+
+ if not self.corpus:
+ self.corpus = list(documents)
+ self.embeddings = self.model.encode(documents)
+ else:
+ self.corpus.extend(documents)
+ new_embeddings = self.model.encode(documents)
+ if self.embeddings is None:
+ self.embeddings = new_embeddings
+ else:
+ self.embeddings = np.vstack([self.embeddings, new_embeddings])
+
+ def reset(self, documents: List[str]) -> None:
+ self.corpus = []
+ self.embeddings = None
+ if documents:
+ self.add_documents(documents)
+
+ def search(self, query: str, k: int = 5) -> List[int]:
+ if not self.corpus or self.embeddings is None:
+ return []
+
+ query_embedding = self.model.encode([query])[0]
+ similarities = cosine_similarity(
+ [query_embedding], self.embeddings
+ )[0]
+ k = min(k, len(self.corpus))
+ top_k_indices = np.argsort(similarities)[-k:][::-1]
+ return top_k_indices.tolist()
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/ace/__init__.py b/labs/AgentStream/exgentic/src/exgentic/agents/ace/__init__.py
new file mode 100644
index 00000000..d9dde690
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/ace/__init__.py
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from .ace_agent import ACEAgent
+from .bulletpoint_analyzer import BulletpointAnalyzer, DEDUP_AVAILABLE
+
+__all__ = ["ACEAgent", "BulletpointAnalyzer", "DEDUP_AVAILABLE"]
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/ace/ace_agent.py b/labs/AgentStream/exgentic/src/exgentic/agents/ace/ace_agent.py
new file mode 100644
index 00000000..094f1b33
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/ace/ace_agent.py
@@ -0,0 +1,88 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+from typing import Any, ClassVar, Optional
+
+from pydantic import ConfigDict
+
+from ...core.agent import Agent
+from ...core.types import ModelSettings
+from ...utils.settings import RunnerName
+
+
+class ACEAgent(Agent):
+
+ display_name: ClassVar[str] = "ACE Agent"
+ slug_name: ClassVar[str] = "ace"
+
+ model_config = ConfigDict(arbitrary_types_allowed=True)
+
+ model: str = "gpt-4o"
+ curator_model: Optional[str] = None
+
+ max_num_rounds: int = 3
+ curator_frequency: int = 1
+ playbook_token_budget: int = 80000
+
+ shuffle_mode: str = "isolated"
+
+ benchmark_id: Optional[str] = None
+
+ initial_playbook: Optional[str] = None
+ initial_playbook_path: Optional[str] = None
+
+ use_json_mode: bool = True
+ runner: RunnerName | None = None
+ model_settings: ModelSettings | None = None
+
+ enable_tool_shortlisting: bool = False
+ max_selected_tools: int = 30
+
+ use_bulletpoint_analyzer: bool = False
+ bulletpoint_analyzer_threshold: float = 0.90
+
+ @classmethod
+ def _get_instance_class(cls):
+ from .ace_instance import ACEAgentInstance
+ return ACEAgentInstance
+
+ @classmethod
+ def _get_instance_class_ref(cls) -> str:
+ return "exgentic.agents.ace.ace_instance:ACEAgentInstance"
+
+ def _get_instance_kwargs(self, session_id: str) -> dict[str, Any]:
+ pb = self.initial_playbook
+ if pb is None and self.initial_playbook_path:
+ with open(self.initial_playbook_path, "r", encoding="utf-8") as fh:
+ pb = fh.read()
+
+ return {
+ "session_id": session_id,
+ "model": self.model,
+ "curator_model": self.curator_model or self.model,
+ "max_num_rounds": self.max_num_rounds,
+ "curator_frequency": self.curator_frequency,
+ "playbook_token_budget": self.playbook_token_budget,
+ "shuffle_mode": self.shuffle_mode,
+ "initial_playbook": pb,
+ "use_json_mode": self.use_json_mode,
+ "model_settings": self.model_settings,
+ "benchmark_id": self.benchmark_id,
+ "use_bulletpoint_analyzer": self.use_bulletpoint_analyzer,
+ "bulletpoint_analyzer_threshold": self.bulletpoint_analyzer_threshold,
+ "enable_tool_shortlisting": self.enable_tool_shortlisting,
+ "max_selected_tools": self.max_selected_tools,
+ }
+
+ @property
+ def model_name(self) -> str:
+ return str(self.model).split("/")[-1]
+
+ def get_models_names(self) -> list[str]:
+ names = [str(self.model)]
+ cm = self.curator_model or self.model
+ if cm != self.model:
+ names.append(str(cm))
+ return names
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/ace/ace_instance.py b/labs/AgentStream/exgentic/src/exgentic/agents/ace/ace_instance.py
new file mode 100644
index 00000000..5850c363
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/ace/ace_instance.py
@@ -0,0 +1,775 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import json
+import re
+import time
+from datetime import datetime
+from typing import Any, Dict, List, Optional, Union
+
+import litellm
+from litellm import (
+ ChatCompletionAssistantMessage,
+ ChatCompletionSystemMessage,
+ ChatCompletionToolMessage,
+ ChatCompletionUserMessage,
+)
+
+from ...core.agent_instance import AgentInstance
+from ...core.types import (
+ Action,
+ ActionType,
+ Message,
+ MessageAction,
+ MessageObservation,
+ MessagePayload,
+ ModelSettings,
+ Observation,
+)
+from ...utils.cost import LiteLLMCostReport
+from ...utils.settings import get_settings
+
+from .playbook_store import PlaybookStore
+from .playbook_utils import (
+ apply_curator_operations,
+ extract_json_from_text,
+ get_playbook_stats,
+ update_bullet_counts,
+)
+from .prompts.curator import CURATOR_PROMPT_NO_GT
+from .prompts.reflector import REFLECTOR_PROMPT_NO_GT
+from .bulletpoint_analyzer import BulletpointAnalyzer, DEDUP_AVAILABLE
+from ..tool_shortlisting import shortlist_tools
+
+try:
+ from ...agents.litellm_tool_calling.utils import ToolsActionsRegistry
+except ImportError:
+ ToolsActionsRegistry = None
+
+_BULLET_ID_RE = re.compile(r"\[([a-z]{2,5}-\d{5})\]")
+
+settings = get_settings()
+
+
+class ACEAgentInstance(AgentInstance):
+
+ def __init__(
+ self,
+ session_id: str,
+ model: str = "gpt-4o",
+ curator_model: str = "gpt-4o",
+ max_num_rounds: int = 3,
+ curator_frequency: int = 1,
+ playbook_token_budget: int = 80000,
+ shuffle_mode: str = "isolated",
+ initial_playbook: Optional[str] = None,
+ use_json_mode: bool = True,
+ model_settings: Optional[ModelSettings] = None,
+ benchmark_id: Optional[str] = None,
+ use_bulletpoint_analyzer: bool = False,
+ bulletpoint_analyzer_threshold: float = 0.90,
+ enable_tool_shortlisting: bool = False,
+ max_selected_tools: int = 30,
+ ) -> None:
+ super().__init__(session_id)
+
+ self.model = model
+ self.curator_model = curator_model
+ self.max_num_rounds = max_num_rounds
+ self.curator_frequency = curator_frequency
+ self.playbook_token_budget = playbook_token_budget
+ self.shuffle_mode = shuffle_mode
+ self.initial_playbook = initial_playbook
+ self.use_json_mode = use_json_mode
+ self.benchmark_id = benchmark_id
+ self.use_bulletpoint_analyzer = use_bulletpoint_analyzer
+ self.bulletpoint_analyzer_threshold = bulletpoint_analyzer_threshold
+ self.enable_tool_shortlisting = enable_tool_shortlisting
+ self.max_selected_tools = max_selected_tools
+
+ if model_settings is None:
+ self._model_settings = ModelSettings()
+ elif isinstance(model_settings, ModelSettings):
+ self._model_settings = model_settings
+ else:
+ self._model_settings = ModelSettings()
+
+ self._cost = LiteLLMCostReport.initialize_empty(model_name=self.model)
+ self._store: Optional[PlaybookStore] = None
+
+ self.messages: list[
+ Union[
+ ChatCompletionAssistantMessage,
+ ChatCompletionToolMessage,
+ ChatCompletionUserMessage,
+ ChatCompletionSystemMessage,
+ ]
+ ] = []
+ self._step_count: int = 0
+
+ self._registry: Optional[ToolsActionsRegistry] = None
+ self._all_actions: list[ActionType] = []
+
+ self._observation_log: List[Dict[str, Any]] = []
+ self._action_log: List[Dict[str, Any]] = []
+
+ def _log_failure(
+ self, component: str, error: Exception, context: Dict[str, Any]
+ ) -> None:
+ try:
+ log_path = self.paths.agent_dir / "ace_failures.jsonl"
+ log_path.parent.mkdir(parents=True, exist_ok=True)
+ entry = {
+ "timestamp": datetime.now().isoformat(),
+ "session_id": self.session_id,
+ "component": component,
+ "error_type": type(error).__name__,
+ "error_message": str(error)[:2000],
+ **{k: str(v)[:2000] if isinstance(v, str) else v
+ for k, v in context.items()},
+ }
+ with open(log_path, "a", encoding="utf-8") as fh:
+ fh.write(json.dumps(entry, ensure_ascii=False) + "\n")
+ except Exception:
+ pass
+
+ def _log_bullet_usage(self, bullet_ids: List[str]) -> None:
+ store = self._store
+ if store is None:
+ return
+ try:
+ log_path = self.paths.agent_dir / "bullet_usage_log.jsonl"
+ log_path.parent.mkdir(parents=True, exist_ok=True)
+
+ from .playbook_utils import extract_playbook_bullets
+ bullets_text = extract_playbook_bullets(store.playbook, bullet_ids)
+
+ entry = {
+ "timestamp": datetime.now().isoformat(),
+ "session_id": self.session_id,
+ "task_id": str(
+ self.context.get("task_id", "") if self.context else ""
+ ),
+ "benchmark_id": self.benchmark_id or "",
+ "store_id": store.store_id,
+ "session_count": store.session_count,
+ "bullet_ids_used": bullet_ids,
+ "bullet_count": len(bullet_ids),
+ "bullets_detail": bullets_text,
+ "total_steps": self._step_count,
+ "total_observations": len(self._observation_log),
+ "total_actions": len(self._action_log),
+ "question_preview": (
+ self.task if self.task else ""
+ ),
+ }
+ with open(log_path, "a", encoding="utf-8") as fh:
+ fh.write(json.dumps(entry, ensure_ascii=False) + "\n")
+ except Exception:
+ pass
+
+ def start(
+ self,
+ task: str,
+ context: Dict[str, Any],
+ actions: list[ActionType],
+ ) -> None:
+ super().start(task, context, actions)
+
+ self._all_actions = list(self.actions)
+ if ToolsActionsRegistry is not None:
+ self._registry = ToolsActionsRegistry(self._all_actions)
+
+ task_group = str(
+ context.get("task_group")
+ or context.get("task_id")
+ or context.get("task_name")
+ or "default"
+ )
+ self._store = PlaybookStore.get_or_create(
+ shuffle_mode=self.shuffle_mode,
+ task_group=task_group,
+ initial_playbook=self.initial_playbook,
+ benchmark_id=self.benchmark_id,
+ )
+ self._store.increment_session()
+
+ playbook = self._store.playbook
+ system_content = self._build_system_prompt(playbook)
+ self._add_message(
+ ChatCompletionSystemMessage(role="system", content=system_content)
+ )
+
+ content_parts: list[Any] = []
+ ctx = ""
+ if self.context:
+ for k, v in self.context.items():
+ if isinstance(v, dict) and v.get("type") == "image_url":
+ content_parts.append({"type": "image_url", "image_url": {"url": v["data"], "detail": "high"}})
+ else:
+ ctx += f"\n<{k}>\n{v}\n"
+
+ text_content = f"{self.task}\n{ctx}"
+ if content_parts:
+ content_parts.insert(0, {"type": "text", "text": text_content})
+ self._add_message(ChatCompletionUserMessage(role="user", content=content_parts))
+ else:
+ self._add_message(
+ ChatCompletionUserMessage(role="user", content=text_content)
+ )
+
+ self.logger.info(
+ "ACE v2 instance started store=%s session_count=%d "
+ "playbook_bullets=%d benchmark=%s tools=%d",
+ self._store.store_id,
+ self._store.session_count,
+ get_playbook_stats(playbook)["total_bullets"],
+ self.benchmark_id or "(none)",
+ len(self._all_actions),
+ )
+
+ def react(self, observation: Optional[Observation]) -> Optional[Action]:
+
+ self._step_count += 1
+ self._observe(observation)
+ self._log_observation(observation)
+
+ tools = self._assistant_tools()
+ response = self._completion(
+ model=self.model,
+ messages=self.messages,
+ tools=tools if tools else None,
+ )
+
+ if response is None:
+ self.logger.error("ACE v2: LLM returned None response")
+ return None
+
+ if response.usage:
+ self._cost.update_cost_from_tokens(
+ response.usage.prompt_tokens,
+ response.usage.completion_tokens,
+ )
+
+ choice = response["choices"][0]
+ message = choice["message"]
+ finish_reason = choice.get("finish_reason")
+
+ if finish_reason == "tool_calls" and self._registry is not None:
+ tool_calls = self._extract_tool_calls(message)
+ self._add_message(
+ ChatCompletionAssistantMessage(
+ role="assistant",
+ tool_calls=[
+ {
+ "id": tc["id"],
+ "type": "function",
+ "function": {
+ "name": tc["name"],
+ "arguments": tc["arguments"],
+ },
+ }
+ for tc in tool_calls
+ ],
+ )
+ )
+ actions = self._registry.tool_calls_to_action(tool_calls)
+
+ for tc in tool_calls:
+ self._action_log.append({
+ "step": self._step_count,
+ "action": tc["name"],
+ "arguments": tc["arguments"],
+ })
+
+ self.logger.info("ACE v2 step %d: tool_calls=%s", self._step_count,
+ [tc["name"] for tc in tool_calls])
+ return actions
+ else:
+ content = message.content if message.content else ""
+ self._add_message(
+ ChatCompletionAssistantMessage(
+ role="assistant", content=content
+ )
+ )
+ self._action_log.append({
+ "step": self._step_count,
+ "action": "message",
+ "content": content,
+ })
+ self.logger.info("ACE v2 step %d: message response", self._step_count)
+ return MessageAction(arguments=Message(content=content))
+
+ def close(self) -> None:
+ store = self._store
+ if store is None:
+ return
+
+ bullet_ids = _BULLET_ID_RE.findall(
+ " ".join(
+ str(m.get("content", "")) if isinstance(m, dict)
+ else str(getattr(m, "content", ""))
+ for m in self.messages
+ if (isinstance(m, dict) and m.get("role") == "assistant")
+ or getattr(m, "role", "") == "assistant"
+ )
+ )
+ self._log_bullet_usage(bullet_ids)
+
+ reflection_content = "(empty)"
+ if self._observation_log:
+ try:
+ reflection_content = self._run_post_session_reflection()
+ except Exception as exc:
+ self.logger.warning(
+ "ACE v2: post-session reflection failed: %s", exc
+ )
+ self._log_failure("reflector", exc, {
+ "observation_count": len(self._observation_log),
+ "action_count": len(self._action_log),
+ })
+
+ if store.session_count % self.curator_frequency == 0:
+ try:
+ self._run_curator(reflection_content)
+ except Exception as exc:
+ self.logger.warning(
+ "ACE v2: curator failed: %s", exc
+ )
+ self._log_failure("curator", exc, {
+ "reflection_preview": reflection_content[:500],
+ })
+
+ store.record_learning(
+ session_id=self.session_id,
+ task_id=str(self.context.get("task_id", "") if self.context else ""),
+ was_correct_before=False,
+ was_correct_after=False,
+ summary=(
+ f"steps={self._step_count} "
+ f"observations={len(self._observation_log)} "
+ f"actions={len(self._action_log)}"
+ ),
+ benchmark_id=self.benchmark_id or "",
+ )
+
+ try:
+ cp = str(self.paths.agent_dir / "playbook_checkpoint.json")
+ store.save_checkpoint(cp)
+ pb = str(self.paths.agent_dir / "playbook.txt")
+ store.save_playbook_text(pb)
+ except Exception as exc:
+ self.logger.warning("ACE v2: failed to save checkpoint: %s", exc)
+
+ def get_cost(self) -> LiteLLMCostReport:
+ return self._cost
+
+ def _build_system_prompt(self, playbook: str) -> str:
+ stats = get_playbook_stats(playbook)
+ has_content = stats["total_bullets"] > 0
+
+ parts = [
+ "You are an expert agent that completes tasks using available tools.",
+ "You have access to a curated playbook of strategies and insights "
+ "learned from previous tasks. Use these to make better decisions.",
+ "",
+ "## Guidelines",
+ "- Read the playbook carefully and apply relevant strategies",
+ "- Pay attention to common mistakes listed and avoid them",
+ "- Use available tools to interact with the environment",
+ "- Think step-by-step before acting",
+ "- When you are confident in your solution, use the finish/submit tool",
+ "- When a playbook bullet influences your decision, mention its ID "
+ "(e.g. [err-00001]) in your reasoning text",
+ ]
+
+ if has_content:
+ parts.extend([
+ "",
+ "## Playbook (accumulated strategies & insights)",
+ "Each line has a bullet ID and usage stats "
+ "(helpful=N means it helped N times, harmful=N means it misled N times).",
+ "Prefer high-helpful, low-harmful bullets.",
+ "",
+ playbook,
+ ])
+ else:
+ parts.extend([
+ "",
+ "## Playbook",
+ "(No strategies accumulated yet. This is the first session.)",
+ ])
+
+ return "\n".join(parts)
+
+ def _add_message(self, message: Any) -> None:
+ self.logger.debug("Adding message: role=%s", getattr(message, "role", "?"))
+ self.messages.append(message)
+
+ def _observe(self, observation: Optional[Observation]) -> None:
+ if observation is None:
+ return
+
+ observations = observation.to_observation_list()
+ if observation.is_empty():
+ if not any(obs.invoking_actions for obs in observations):
+ return
+
+ for obs in observations:
+ if isinstance(obs, MessageObservation) and isinstance(
+ obs.result, MessagePayload
+ ):
+ self._add_message(
+ ChatCompletionUserMessage(
+ role="user", content=obs.result.message
+ )
+ )
+ continue
+
+ if len(obs.invoking_actions) > 0:
+ invoking = obs.invoking_actions[0]
+ if invoking.name == "message":
+ self._add_message(
+ ChatCompletionUserMessage(
+ role="user", content=str(obs)
+ )
+ )
+ continue
+
+ action_id = invoking.id
+ tool_call_id = invoking.id
+ if not (
+ isinstance(tool_call_id, str)
+ and tool_call_id.startswith("call_")
+ ):
+ if self._registry is not None:
+ tool_call_id = (
+ self._registry.action_id_to_tool_call_id.get(
+ action_id, tool_call_id
+ )
+ )
+
+ value = obs.result
+ try:
+ content = json.dumps(
+ value, ensure_ascii=False, separators=(",", ":")
+ )
+ except TypeError:
+ content = str(value)
+
+ if tool_call_id is not None:
+ self._add_message(
+ ChatCompletionToolMessage(
+ role="tool",
+ tool_call_id=tool_call_id,
+ content=content,
+ )
+ )
+ else:
+ self._add_message(
+ ChatCompletionUserMessage(
+ role="user",
+ content=f"Tool result: {content}",
+ )
+ )
+ else:
+ self._add_message(
+ ChatCompletionUserMessage(
+ role="user", content=str(obs)
+ )
+ )
+
+ def _log_observation(self, observation: Optional[Observation]) -> None:
+ if observation is None or observation.is_empty():
+ return
+
+ for obs in observation.to_observation_list():
+ result = obs.result
+ if result is None:
+ continue
+
+ entry: Dict[str, Any] = {"step": self._step_count}
+ if isinstance(result, str):
+ entry["content"] = result
+ elif isinstance(result, dict):
+ entry["content"] = json.dumps(result, ensure_ascii=False)
+ else:
+ entry["content"] = str(result)
+
+ if obs.invoking_actions:
+ entry["action"] = obs.invoking_actions[0].name
+
+ self._observation_log.append(entry)
+
+ def _assistant_tools(self) -> list[dict[str, Any]]:
+ if self._registry is None:
+ return []
+ tools = self._registry.openai_tools()
+ if not self.enable_tool_shortlisting:
+ return tools
+
+ def _cost_cb(usage):
+ if usage:
+ self._cost.update_cost_from_tokens(
+ usage.prompt_tokens, usage.completion_tokens
+ )
+
+ return shortlist_tools(
+ tools=tools,
+ max_selected=self.max_selected_tools,
+ messages=self.messages,
+ completion_fn=self._completion,
+ model=self.model,
+ logger=self.logger,
+ cost_callback=_cost_cb,
+ )
+
+ @staticmethod
+ def _extract_tool_calls(message: Any) -> list[dict[str, str]]:
+ if not hasattr(message, "tool_calls") or not message.tool_calls:
+ return []
+ tool_calls = []
+ for tc in message.tool_calls:
+ tool_calls.append({
+ "name": tc.function.name,
+ "arguments": tc.function.arguments,
+ "id": tc.id,
+ })
+ return tool_calls
+
+ def _completion(self, **kwargs) -> Any:
+ call_kwargs = self._model_settings.model_dump(
+ exclude_none=True,
+ exclude={"num_retries", "retry_after", "retry_strategy"},
+ )
+ call_kwargs.update(kwargs)
+ if call_kwargs.get("tools") is None:
+ call_kwargs.pop("tools", None)
+
+ max_attempts = 3
+ for attempt in range(max_attempts):
+ try:
+ response = litellm.completion(**call_kwargs)
+ return response
+ except Exception as exc:
+ self.logger.warning(
+ "ACE LLM call attempt %d/%d failed: %s",
+ attempt + 1,
+ max_attempts,
+ exc,
+ )
+ if attempt + 1 >= max_attempts:
+ raise
+ time.sleep(2 ** attempt)
+ return None
+
+ def _llm_call_simple(
+ self,
+ model: str,
+ prompt: str,
+ *,
+ json_mode: bool = False,
+ ) -> str:
+ kwargs: Dict[str, Any] = {
+ "model": model,
+ "messages": [{"role": "user", "content": prompt}],
+ "temperature": 0.0,
+ }
+ if json_mode:
+ kwargs["response_format"] = {"type": "json_object"}
+
+ max_attempts = 3
+ for attempt in range(max_attempts):
+ try:
+ resp = litellm.completion(**kwargs)
+ if resp.usage:
+ self._cost.update_cost_from_tokens(
+ resp.usage.prompt_tokens,
+ resp.usage.completion_tokens,
+ )
+ content = resp.choices[0].message.content
+ if content is None:
+ raise ValueError("LLM returned None content")
+ return content
+ except Exception as exc:
+ self.logger.warning(
+ "ACE simple LLM call attempt %d/%d failed: %s",
+ attempt + 1,
+ max_attempts,
+ exc,
+ )
+ if attempt + 1 >= max_attempts:
+ self._log_failure(
+ "llm_call", exc, {
+ "model": model,
+ "prompt_length": len(prompt),
+ "prompt_preview": prompt[:500],
+ "attempts": max_attempts,
+ },
+ )
+ raise
+ time.sleep(2 ** attempt)
+ return ""
+
+ def _run_post_session_reflection(self) -> str:
+ store = self._store
+ assert store is not None
+
+ playbook = store.playbook
+ session_trace = self._build_session_trace()
+
+ prompt = REFLECTOR_PROMPT_NO_GT.format(
+ question=self.task if self.task else "",
+ reasoning_trace=session_trace,
+ predicted_answer="(see session trace above)",
+ bullets_used=playbook,
+ )
+
+ raw = self._llm_call_simple(
+ self.model, prompt, json_mode=self.use_json_mode
+ )
+
+ bullet_tags: List[Dict] = []
+ reflection_text = raw
+ parsed = extract_json_from_text(raw)
+ if parsed and isinstance(parsed, dict):
+ bullet_tags = parsed.get("bullet_tags", [])
+ reflection_text = parsed.get("reasoning", raw)
+ else:
+ self.logger.warning(
+ "ACE Reflector: JSON parse failed, raw length=%d", len(raw)
+ )
+ self._log_failure(
+ "reflector_parse", ValueError("JSON parse failed"), {
+ "raw_response_preview": raw[:1000],
+ },
+ )
+
+ if bullet_tags:
+ store.playbook = update_bullet_counts(store.playbook, bullet_tags)
+
+ self.logger.info(
+ "ACE post-session reflection: %d bullet tags updated",
+ len(bullet_tags),
+ )
+
+ return reflection_text
+
+ def _build_session_trace(self) -> str:
+ events: List[Dict[str, Any]] = []
+ for entry in self._action_log:
+ events.append({"type": "action", **entry})
+ for entry in self._observation_log:
+ events.append({"type": "observation", **entry})
+
+ events.sort(key=lambda e: (e.get("step", 0), 0 if e["type"] == "action" else 1))
+
+ lines: List[str] = []
+ for event in events:
+ step = event.get("step", "?")
+ if event["type"] == "action":
+ action = event.get("action", "?")
+ args = event.get("arguments", event.get("content", ""))
+ lines.append(f"[Step {step}] Action: {action}")
+ if args:
+ lines.append(f" Args: {str(args)}")
+ else:
+ action = event.get("action", "env")
+ content = event.get("content", "")
+ lines.append(f"[Step {step}] Observation from {action}:")
+ lines.append(f" {content}")
+
+ return "\n".join(lines) if lines else "(No session trace recorded)"
+
+ def _run_curator(self, reflection_content: str) -> None:
+ store = self._store
+ assert store is not None
+
+ playbook = store.playbook
+ stats = get_playbook_stats(playbook)
+
+ question_context = self.task if self.task else ""
+ if self.context:
+ question_context += "".join(
+ f"\n<{k}>\n{v}\n"
+ for k, v in self.context.items()
+ )
+
+ prompt = CURATOR_PROMPT_NO_GT.format(
+ token_budget=self.playbook_token_budget,
+ current_step=store.session_count,
+ total_samples="ongoing",
+ playbook_stats=json.dumps(stats, indent=2),
+ recent_reflection=reflection_content,
+ current_playbook=playbook,
+ question_context=question_context,
+ )
+
+ raw = self._llm_call_simple(
+ self.curator_model, prompt, json_mode=self.use_json_mode
+ )
+
+ if raw.startswith("INCORRECT_DUE_TO_EMPTY_RESPONSE"):
+ self.logger.warning("ACE Curator: skipping due to empty response")
+ self._log_failure(
+ "curator_empty_response", ValueError("empty LLM response"), {
+ "session_count": store.session_count,
+ },
+ )
+ return
+
+ parsed = extract_json_from_text(raw)
+ if parsed and isinstance(parsed, dict):
+ if "operations" not in parsed or not isinstance(parsed["operations"], list):
+ self.logger.warning("ACE Curator: missing or invalid 'operations' field")
+ self._log_failure(
+ "curator_schema", ValueError("missing 'operations' list"), {
+ "raw_response_preview": raw[:1000],
+ "parsed_keys": list(parsed.keys()),
+ },
+ )
+ return
+
+ ops = parsed["operations"]
+ valid_ops = []
+ for op in ops:
+ if not isinstance(op, dict) or "type" not in op:
+ continue
+ if op["type"] == "ADD":
+ if "section" in op and "content" in op:
+ valid_ops.append(op)
+ else:
+ valid_ops.append(op)
+
+ if valid_ops:
+ new_playbook, new_id = apply_curator_operations(
+ playbook, valid_ops, store.next_global_id
+ )
+ store.playbook = new_playbook
+ store.next_global_id = new_id
+ self.logger.info("ACE Curator: applied %d operations", len(valid_ops))
+
+ if self.use_bulletpoint_analyzer and DEDUP_AVAILABLE:
+ self.logger.info(
+ "ACE BulletpointAnalyzer: running (threshold=%.2f)",
+ self.bulletpoint_analyzer_threshold,
+ )
+ analyzer = BulletpointAnalyzer(
+ llm_merge_fn=lambda p: self._llm_call_simple(
+ self.curator_model, p, json_mode=False
+ ),
+ )
+ store.playbook = analyzer.analyze(
+ playbook=store.playbook,
+ threshold=self.bulletpoint_analyzer_threshold,
+ merge=True,
+ )
+ else:
+ self.logger.warning("ACE Curator: failed to parse response")
+ self._log_failure(
+ "curator_parse", ValueError("JSON parse failed"), {
+ "raw_response_preview": raw[:1000],
+ },
+ )
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/ace/bulletpoint_analyzer.py b/labs/AgentStream/exgentic/src/exgentic/agents/ace/bulletpoint_analyzer.py
new file mode 100644
index 00000000..30636587
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/ace/bulletpoint_analyzer.py
@@ -0,0 +1,200 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import re
+from typing import Any, Dict, List, Optional, Tuple
+
+try:
+ import numpy as np
+ from sentence_transformers import SentenceTransformer
+ import faiss
+
+ DEDUP_AVAILABLE = True
+except ImportError:
+ DEDUP_AVAILABLE = False
+ np = None
+
+from .playbook_utils import parse_playbook_line, format_playbook_line
+
+
+class BulletpointAnalyzer:
+
+ def __init__(
+ self,
+ llm_merge_fn: Optional[Any] = None,
+ embedding_model_name: str = "all-mpnet-base-v2",
+ ) -> None:
+ self.llm_merge_fn = llm_merge_fn
+ self.embedding_model_name = embedding_model_name
+ self._embedding_model: Optional[Any] = None
+
+ def _load_embedding_model(self) -> None:
+ if self._embedding_model is None and DEDUP_AVAILABLE:
+ self._embedding_model = SentenceTransformer(self.embedding_model_name)
+
+ @staticmethod
+ def _parse_playbook(
+ playbook: str,
+ ) -> Tuple[List[str], List[Dict[str, Any]], Dict[int, int]]:
+ lines = playbook.strip().split("\n")
+ bullets: List[Dict[str, Any]] = []
+ bullet_line_mapping: Dict[int, int] = {}
+
+ for line_idx, line in enumerate(lines):
+ parsed = parse_playbook_line(line)
+ if parsed:
+ parsed["line_number"] = line_idx + 1
+ parsed["original_line"] = line
+ bullet_index = len(bullets)
+ bullet_line_mapping[bullet_index] = line_idx
+ bullets.append(parsed)
+
+ return lines, bullets, bullet_line_mapping
+
+ def _compute_embeddings(self, bullets: List[Dict[str, Any]]) -> Any:
+ if not DEDUP_AVAILABLE:
+ raise RuntimeError("Cannot compute embeddings without sentence-transformers")
+ self._load_embedding_model()
+ contents = [b["content"] for b in bullets]
+ embeddings = self._embedding_model.encode(
+ contents, convert_to_numpy=True, show_progress_bar=False
+ )
+ faiss.normalize_L2(embeddings)
+ return embeddings
+
+ @staticmethod
+ def _find_similar_groups(
+ bullets: List[Dict[str, Any]],
+ embeddings: Any,
+ threshold: float,
+ ) -> List[Dict[str, Any]]:
+ similarity_matrix = np.dot(embeddings, embeddings.T)
+ duplicate_groups: List[Dict[str, Any]] = []
+ visited: set[int] = set()
+
+ for i in range(len(bullets)):
+ if i in visited:
+ continue
+ similar_indices = []
+ for j in range(i + 1, len(bullets)):
+ if similarity_matrix[i, j] >= threshold:
+ similar_indices.append(j)
+ if similar_indices:
+ group = [i] + similar_indices
+ duplicate_groups.append(
+ {"indices": group, "bullets": [bullets[idx] for idx in group]}
+ )
+ visited.update(group)
+
+ return duplicate_groups
+
+ def _merge_bullets_with_llm(
+ self, bullets_group: List[Dict[str, Any]]
+ ) -> Optional[Dict[str, Any]]:
+ if len(bullets_group) == 1:
+ return bullets_group[0]
+
+ if self.llm_merge_fn is None:
+ return bullets_group[0]
+
+ bullets_text = "\n".join(
+ f"{i+1}. [{b['id']}] helpful={b['helpful']} harmful={b['harmful']} :: {b['content']}"
+ for i, b in enumerate(bullets_group)
+ )
+ total_helpful = sum(b["helpful"] for b in bullets_group)
+ total_harmful = sum(b["harmful"] for b in bullets_group)
+ base_id = bullets_group[0]["id"]
+
+ prompt = (
+ f"You are merging similar playbook bulletpoints into a single, "
+ f"comprehensive entry.\n\n"
+ f"Given these similar bulletpoints:\n{bullets_text}\n\n"
+ f"Merge them into ONE bulletpoint that captures all important "
+ f"information while removing redundancy.\n\n"
+ f"Requirements:\n"
+ f"1. Keep the ID from the first entry: [{base_id}]\n"
+ f"2. Use combined counts: helpful={total_helpful} harmful={total_harmful}\n"
+ f"3. Combine the content to be comprehensive but concise\n"
+ f"4. Output ONLY in this format: [{base_id}] helpful={total_helpful} "
+ f"harmful={total_harmful} :: [merged content]\n\n"
+ f"Do NOT include any explanation, just output the merged bulletpoint."
+ )
+
+ try:
+ merged_content = self.llm_merge_fn(prompt).strip()
+ pattern = r"\[([^\]]+)\]\s+helpful=(\d+)\s+harmful=(\d+)\s+::\s+(.+)"
+ match = re.match(pattern, merged_content)
+ if match:
+ bullet_id, helpful, harmful, content = match.groups()
+ return {
+ "id": bullet_id,
+ "helpful": int(helpful),
+ "harmful": int(harmful),
+ "content": content.strip(),
+ "original_line": format_playbook_line(
+ bullet_id, int(helpful), int(harmful), content.strip()
+ ),
+ "is_merged": True,
+ "original_count": len(bullets_group),
+ }
+ else:
+ return bullets_group[0]
+ except Exception:
+ return bullets_group[0]
+
+ def analyze(
+ self,
+ playbook: str,
+ threshold: float = 0.90,
+ merge: bool = True,
+ ) -> str:
+ if not DEDUP_AVAILABLE:
+ return playbook
+
+ original_lines, bullets, bullet_line_mapping = self._parse_playbook(playbook)
+
+ if len(bullets) == 0:
+ return playbook
+
+ embeddings = self._compute_embeddings(bullets)
+ duplicate_groups = self._find_similar_groups(bullets, embeddings, threshold)
+
+ if len(duplicate_groups) == 0:
+ return playbook
+
+ merge_mapping: Dict[int, Dict[str, Any]] = {}
+ processed_indices: set[int] = set()
+
+ if merge:
+ for group in duplicate_groups:
+ indices = group["indices"]
+ merged_bullet = self._merge_bullets_with_llm(group["bullets"])
+ if merged_bullet:
+ merge_mapping[indices[0]] = merged_bullet
+ processed_indices.update(indices)
+ else:
+ for group in duplicate_groups:
+ indices = group["indices"]
+ processed_indices.update(indices[1:])
+
+ output_lines: List[str] = []
+ for line_idx, original_line in enumerate(original_lines):
+ current_bullet_idx = None
+ for bi, li in bullet_line_mapping.items():
+ if li == line_idx:
+ current_bullet_idx = bi
+ break
+
+ if current_bullet_idx is not None:
+ if current_bullet_idx in merge_mapping:
+ output_lines.append(merge_mapping[current_bullet_idx]["original_line"])
+ elif current_bullet_idx in processed_indices:
+ continue
+ else:
+ output_lines.append(original_line)
+ else:
+ output_lines.append(original_line)
+
+ return "\n".join(output_lines)
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/ace/playbook_store.py b/labs/AgentStream/exgentic/src/exgentic/agents/ace/playbook_store.py
new file mode 100644
index 00000000..61bf26ce
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/ace/playbook_store.py
@@ -0,0 +1,172 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import json
+import threading
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Dict, List, Optional
+
+
+DEFAULT_PLAYBOOK = """\
+## STRATEGIES & INSIGHTS
+
+## FORMULAS & CALCULATIONS
+
+## CODE SNIPPETS & TEMPLATES
+
+## COMMON MISTAKES TO AVOID
+
+## PROBLEM-SOLVING HEURISTICS
+
+## CONTEXT CLUES & INDICATORS
+
+## OTHERS"""
+
+
+@dataclass
+class LearningEvent:
+ session_id: str
+ task_id: str
+ step: int
+ was_correct_before: bool
+ was_correct_after: bool
+ summary: str
+ benchmark_id: str = ""
+
+
+class PlaybookStore:
+
+ _instances: Dict[str, "PlaybookStore"] = {}
+ _global_lock = threading.Lock()
+
+ @classmethod
+ def get_or_create(
+ cls,
+ shuffle_mode: str = "isolated",
+ task_group: Optional[str] = None,
+ initial_playbook: Optional[str] = None,
+ benchmark_id: Optional[str] = None,
+ ) -> "PlaybookStore":
+ if shuffle_mode == "isolated":
+ bm = benchmark_id or task_group or "default"
+ key = f"ace_isolated_{bm}"
+ elif shuffle_mode == "sequential":
+ key = "ace_sequential_global"
+ elif shuffle_mode == "interleaved":
+ key = "ace_interleaved_global"
+ else:
+ raise ValueError(f"Unknown shuffle_mode: {shuffle_mode!r}")
+
+ with cls._global_lock:
+ if key not in cls._instances:
+ cls._instances[key] = cls(
+ store_id=key,
+ initial_playbook=initial_playbook or DEFAULT_PLAYBOOK,
+ )
+ return cls._instances[key]
+
+ @classmethod
+ def reset_all(cls) -> None:
+ with cls._global_lock:
+ cls._instances.clear()
+
+ @classmethod
+ def list_stores(cls) -> Dict[str, "PlaybookStore"]:
+ with cls._global_lock:
+ return dict(cls._instances)
+
+ def __init__(self, store_id: str, initial_playbook: str) -> None:
+ self.store_id = store_id
+ self._lock = threading.Lock()
+ self._playbook: str = initial_playbook
+ self._next_global_id: int = 1
+ self._session_count: int = 0
+ self._history: List[LearningEvent] = []
+ self._benchmark_counts: Dict[str, int] = {}
+
+ @property
+ def playbook(self) -> str:
+ with self._lock:
+ return self._playbook
+
+ @playbook.setter
+ def playbook(self, value: str) -> None:
+ with self._lock:
+ self._playbook = value
+
+ @property
+ def next_global_id(self) -> int:
+ with self._lock:
+ return self._next_global_id
+
+ @next_global_id.setter
+ def next_global_id(self, value: int) -> None:
+ with self._lock:
+ self._next_global_id = value
+
+ @property
+ def session_count(self) -> int:
+ with self._lock:
+ return self._session_count
+
+ def increment_session(self) -> int:
+ with self._lock:
+ self._session_count += 1
+ return self._session_count
+
+ def record_learning(
+ self,
+ session_id: str,
+ task_id: str,
+ was_correct_before: bool,
+ was_correct_after: bool,
+ summary: str,
+ benchmark_id: str = "",
+ ) -> None:
+ with self._lock:
+ self._history.append(
+ LearningEvent(
+ session_id=session_id,
+ task_id=task_id,
+ step=self._session_count,
+ was_correct_before=was_correct_before,
+ was_correct_after=was_correct_after,
+ summary=summary[:500],
+ benchmark_id=benchmark_id,
+ )
+ )
+ if benchmark_id:
+ self._benchmark_counts[benchmark_id] = (
+ self._benchmark_counts.get(benchmark_id, 0) + 1
+ )
+
+ def save_checkpoint(self, path: str) -> None:
+ with self._lock:
+ payload = {
+ "store_id": self.store_id,
+ "playbook": self._playbook,
+ "next_global_id": self._next_global_id,
+ "session_count": self._session_count,
+ "history_len": len(self._history),
+ "benchmark_counts": dict(self._benchmark_counts),
+ }
+ Path(path).parent.mkdir(parents=True, exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ json.dump(payload, fh, indent=2, ensure_ascii=False)
+
+ def load_checkpoint(self, path: str) -> None:
+ with open(path, "r", encoding="utf-8") as fh:
+ payload = json.load(fh)
+ with self._lock:
+ self._playbook = payload["playbook"]
+ self._next_global_id = payload["next_global_id"]
+ self._session_count = payload.get("session_count", 0)
+ self._benchmark_counts = payload.get("benchmark_counts", {})
+
+ def save_playbook_text(self, path: str) -> None:
+ Path(path).parent.mkdir(parents=True, exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ fh.write(self.playbook)
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/ace/playbook_utils.py b/labs/AgentStream/exgentic/src/exgentic/agents/ace/playbook_utils.py
new file mode 100644
index 00000000..fa1f915b
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/ace/playbook_utils.py
@@ -0,0 +1,230 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import json
+import re
+from typing import Any, Dict, List, Optional, Tuple
+
+_SLUG_MAP = {
+ "strategies_and_insights": "str",
+ "formulas_and_calculations": "calc",
+ "code_snippets_and_templates": "code",
+ "common_mistakes_to_avoid": "err",
+ "problem_solving_heuristics": "prob",
+ "context_clues_and_indicators": "ctx",
+ "others": "misc",
+ "meta_strategies": "meta",
+}
+
+
+def get_section_slug(section_name: str) -> str:
+ clean = section_name.lower().strip().replace(" ", "_").replace("&", "and")
+ if clean in _SLUG_MAP:
+ return _SLUG_MAP[clean]
+ words = clean.split("_")
+ if len(words) == 1:
+ return words[0][:4]
+ return "".join(w[0] for w in words[:5])
+
+
+_LINE_RE = re.compile(
+ r"\[([^\]]+)\]\s*helpful=(\d+)\s*harmful=(\d+)\s*::\s*(.*)"
+)
+
+
+def parse_playbook_line(line: str) -> Optional[Dict[str, Any]]:
+ m = _LINE_RE.match(line.strip())
+ if m:
+ return {
+ "id": m.group(1),
+ "helpful": int(m.group(2)),
+ "harmful": int(m.group(3)),
+ "content": m.group(4),
+ "raw_line": line,
+ }
+ return None
+
+
+def format_playbook_line(
+ bullet_id: str, helpful: int, harmful: int, content: str
+) -> str:
+ return f"[{bullet_id}] helpful={helpful} harmful={harmful} :: {content}"
+
+def update_bullet_counts(playbook_text: str, bullet_tags: List[Dict]) -> str:
+ tag_map: Dict[str, str] = {}
+ for tag in bullet_tags:
+ if not isinstance(tag, dict):
+ continue
+ bid = tag.get("id") or tag.get("bullet", "")
+ tval = tag.get("tag", "neutral")
+ if bid:
+ tag_map[bid] = tval
+
+ if not tag_map:
+ return playbook_text
+
+ lines = playbook_text.split("\n")
+ updated: List[str] = []
+ for line in lines:
+ parsed = parse_playbook_line(line)
+ if parsed and parsed["id"] in tag_map:
+ t = tag_map[parsed["id"]]
+ if t == "helpful":
+ parsed["helpful"] += 1
+ elif t == "harmful":
+ parsed["harmful"] += 1
+ updated.append(
+ format_playbook_line(
+ parsed["id"], parsed["helpful"], parsed["harmful"], parsed["content"]
+ )
+ )
+ else:
+ updated.append(line)
+ return "\n".join(updated)
+
+
+def apply_curator_operations(
+ playbook_text: str,
+ operations: List[Dict[str, Any]],
+ next_id: int,
+) -> Tuple[str, int]:
+ lines = playbook_text.split("\n")
+ sections: Dict[str, int] = {}
+ for i, line in enumerate(lines):
+ if line.strip().startswith("##"):
+ header = line.strip()[2:].strip()
+ norm = header.lower().replace(" ", "_").replace("&", "and")
+ sections[norm] = i
+
+ bullets_to_add: List[Tuple[str, str]] = []
+
+ for op in operations:
+ if op.get("type") != "ADD":
+ continue
+ section_raw = op.get("section", "others")
+ section_norm = section_raw.lower().replace(" ", "_").replace("&", "and")
+ if section_norm not in sections:
+ section_norm = "others"
+
+ slug = get_section_slug(section_norm)
+ new_id = f"{slug}-{next_id:05d}"
+ next_id += 1
+ content = op.get("content", "")
+ new_line = format_playbook_line(new_id, 0, 0, content)
+ bullets_to_add.append((section_norm, new_line))
+
+ final: List[str] = []
+ current_section: Optional[str] = None
+
+ for line in lines:
+ if line.strip().startswith("##"):
+ if current_section is not None:
+ for sec, bline in bullets_to_add:
+ if sec == current_section:
+ final.append(bline)
+ bullets_to_add = [
+ (s, b) for s, b in bullets_to_add if s != current_section
+ ]
+ header = line.strip()[2:].strip()
+ current_section = header.lower().replace(" ", "_").replace("&", "and")
+ final.append(line)
+
+ if current_section is not None:
+ for sec, bline in bullets_to_add:
+ if sec == current_section:
+ final.append(bline)
+ bullets_to_add = [(s, b) for s, b in bullets_to_add if s != current_section]
+
+ for _, bline in bullets_to_add:
+ final.append(bline)
+
+ return "\n".join(final), next_id
+
+def get_playbook_stats(playbook_text: str) -> Dict[str, Any]:
+ stats: Dict[str, Any] = {
+ "total_bullets": 0,
+ "high_performing": 0,
+ "problematic": 0,
+ "unused": 0,
+ "by_section": {},
+ }
+ current_section = "general"
+ for line in playbook_text.split("\n"):
+ if line.strip().startswith("##"):
+ current_section = line.strip()[2:].strip()
+ continue
+ parsed = parse_playbook_line(line)
+ if parsed:
+ stats["total_bullets"] += 1
+ h, d = parsed["helpful"], parsed["harmful"]
+ if h > 5 and d < 2:
+ stats["high_performing"] += 1
+ elif d >= h and d > 0:
+ stats["problematic"] += 1
+ elif h + d == 0:
+ stats["unused"] += 1
+ sec = stats["by_section"].setdefault(
+ current_section, {"count": 0, "helpful": 0, "harmful": 0}
+ )
+ sec["count"] += 1
+ sec["helpful"] += h
+ sec["harmful"] += d
+ return stats
+
+def extract_playbook_bullets(
+ playbook_text: str, bullet_ids: List[str]
+) -> str:
+ if not bullet_ids:
+ return "(No bullets used by generator)"
+ found: List[str] = []
+ for line in playbook_text.split("\n"):
+ parsed = parse_playbook_line(line)
+ if parsed and parsed["id"] in bullet_ids:
+ found.append(
+ format_playbook_line(
+ parsed["id"], parsed["helpful"], parsed["harmful"], parsed["content"]
+ )
+ )
+ return "\n".join(found) if found else "(No matching bullets found)"
+
+
+def extract_json_from_text(text: str) -> Optional[Dict[str, Any]]:
+ try:
+ return json.loads(text.strip())
+ except json.JSONDecodeError:
+ pass
+
+ for m in re.finditer(r"```json\s*(.*?)\s*```", text, re.DOTALL | re.I):
+ try:
+ return json.loads(m.group(1).strip())
+ except json.JSONDecodeError:
+ continue
+
+ i = 0
+ while i < len(text):
+ if text[i] == "{":
+ depth, start = 1, i
+ i += 1
+ while i < len(text) and depth > 0:
+ if text[i] == "{":
+ depth += 1
+ elif text[i] == "}":
+ depth -= 1
+ elif text[i] == '"':
+ i += 1
+ while i < len(text) and text[i] != '"':
+ if text[i] == "\\":
+ i += 1
+ i += 1
+ i += 1
+ if depth == 0:
+ try:
+ return json.loads(text[start:i])
+ except json.JSONDecodeError:
+ pass
+ else:
+ i += 1
+
+ return None
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/ace/prompts/__init__.py b/labs/AgentStream/exgentic/src/exgentic/agents/ace/prompts/__init__.py
new file mode 100644
index 00000000..125af47f
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/ace/prompts/__init__.py
@@ -0,0 +1,10 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from .reflector import REFLECTOR_PROMPT_NO_GT
+from .curator import CURATOR_PROMPT_NO_GT
+
+__all__ = [
+ "REFLECTOR_PROMPT_NO_GT",
+ "CURATOR_PROMPT_NO_GT",
+]
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/ace/prompts/curator.py b/labs/AgentStream/exgentic/src/exgentic/agents/ace/prompts/curator.py
new file mode 100644
index 00000000..02c217ff
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/ace/prompts/curator.py
@@ -0,0 +1,67 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+CURATOR_PROMPT_NO_GT = """\
+You are a master curator of knowledge. Your job is to identify what new insights \
+should be added to an existing playbook based on a reflection from a previous attempt.
+
+**Context:**
+- The playbook you created will be used to help answering similar questions.
+- The reflection is generated using environment feedback that will NOT be available \
+when the playbook is being used.
+
+**CRITICAL: You MUST respond with valid JSON only. Do not use markdown formatting or code blocks.**
+
+**Instructions:**
+- Review the existing playbook and the reflection from the previous attempt
+- Identify ONLY the NEW insights, strategies, or mistakes that are MISSING from the current playbook
+- Avoid redundancy - if similar advice already exists, only add new content that is a perfect complement to the existing playbook
+- Do NOT regenerate the entire playbook - only provide the additions needed
+- Focus on quality over quantity - a focused, well-organized playbook is better than an exhaustive one
+- Format your response as a PURE JSON object with specific sections
+- For any operation if no new content to add, return an empty list for the operations field
+- Be concise and specific - each addition should be actionable
+
+
+**Training Context:**
+- Total token budget: {token_budget} tokens
+- Training progress: Sample {current_step} out of {total_samples}
+
+**Current Playbook Stats:**
+{playbook_stats}
+
+**Recent Reflection:**
+{recent_reflection}
+
+**Current Playbook:**
+{current_playbook}
+
+**Question Context:**
+{question_context}
+
+**Your Task:**
+Output ONLY a valid JSON object with these exact fields:
+- reasoning: your chain of thought / reasoning / thinking process
+- operations: a list of operations to be performed on the playbook
+ - type: the type of operation to be performed
+ - section: the section to add the bullet to
+ - content: the new content of the bullet
+
+**Available Operations:**
+1. ADD: Create new bullet points with fresh IDs
+ - section: the section to add the new bullet to
+ - content: the new content of the bullet. Note: no need to include the bullet_id \
+in the content like '[ctx-00263] helpful=1 harmful=0 ::', the bullet_id will be added by the system.
+
+**RESPONSE FORMAT - Output ONLY this JSON structure (no markdown, no code blocks):**
+{{
+ "reasoning": "[Your reasoning here]",
+ "operations": [
+ {{
+ "type": "ADD",
+ "section": "formulas_and_calculations",
+ "content": "[New calculation method...]"
+ }}
+ ]
+}}
+"""
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/ace/prompts/reflector.py b/labs/AgentStream/exgentic/src/exgentic/agents/ace/prompts/reflector.py
new file mode 100644
index 00000000..e2b3a2e0
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/ace/prompts/reflector.py
@@ -0,0 +1,53 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+REFLECTOR_PROMPT_NO_GT = """\
+You are an expert analyst and educator. Your job is to analyze a model's \
+reasoning process and identify potential issues or strengths based on the \
+reasoning trace alone.
+
+**Instructions:**
+- Carefully analyze the model's reasoning trace to evaluate its approach
+- The reasoning trace includes both the model's actions and environment observations in chronological order
+- Identify potential conceptual errors, calculation mistakes, or misapplied strategies
+- Also note what the model did well
+- Provide actionable insights that could help the model perform better in future tasks
+- Focus on the root cause, not just surface-level observations
+- Be specific about what could be improved
+- You will receive the full playbook that was available to the agent.
+- Based on the reasoning trace, infer which bullets the agent likely applied or was influenced by, and tag each relevant bullet as 'helpful', 'harmful', or 'neutral'. Skip unrelated bullets.
+
+Your output should be a json object, which contains the following fields
+ - reasoning: your chain of thought / reasoning / thinking process, detailed analysis and calculations
+ - error_identification: what potential issues exist in the reasoning? (or "none identified" if the approach appears sound)
+ - root_cause_analysis: why might these issues occur? What concept may have been misunderstood?
+ - correct_approach: what could the model do differently or better?
+ - key_insight: what strategy, formula, or principle should be remembered for future tasks?
+ - bullet_tags: a list of json objects with bullet id and tag for each relevant playbook bullet
+
+
+**Question:**
+{question}
+
+**Model's Reasoning Trace:**
+{reasoning_trace}
+
+**Model's Predicted Answer:**
+{predicted_answer}
+
+**Full Playbook:**
+{bullets_used}
+
+**Answer in this exact JSON format:**
+{{
+ "reasoning": "[Your chain of thought / reasoning / thinking process]",
+ "error_identification": "[What potential issues exist in the reasoning?]",
+ "root_cause_analysis": "[Why might these issues occur?]",
+ "correct_approach": "[What could the model do differently or better?]",
+ "key_insight": "[What strategy or principle should be remembered?]",
+ "bullet_tags": [
+ {{"id": "calc-00001", "tag": "helpful"}},
+ {{"id": "fin-00002", "tag": "harmful"}}
+ ]
+}}
+"""
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/__init__.py b/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/__init__.py
new file mode 100644
index 00000000..17ea7313
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/__init__.py
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from .autoskill_agent import AutoSkillAgent
+from .skill_store import SkillStore
+
+__all__ = ["AutoSkillAgent", "SkillStore"]
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/autoskill_agent.py b/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/autoskill_agent.py
new file mode 100644
index 00000000..ec0d498a
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/autoskill_agent.py
@@ -0,0 +1,81 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+from typing import Any, ClassVar, Optional
+
+from pydantic import ConfigDict
+
+from ...core.agent import Agent
+from ...core.types import ModelSettings
+from ...utils.settings import RunnerName
+
+
+class AutoSkillAgent(Agent):
+
+ display_name: ClassVar[str] = "AutoSkill Agent"
+ slug_name: ClassVar[str] = "autoskill"
+
+ model_config = ConfigDict(arbitrary_types_allowed=True)
+
+
+ model: str = "gpt-4o"
+ skill_model: Optional[str] = None
+
+ retrieve_k: int = 3
+ retrieval_threshold: float = 0.4
+ bm25_weight: float = 0.1
+ dedupe_similarity_threshold: float = 0.4
+ embedding_model: str = "all-MiniLM-L6-v2"
+ enable_query_rewrite: bool = True
+ max_context_chars: int = 6000
+
+ shuffle_mode: str = "isolated"
+
+ benchmark_id: Optional[str] = None
+
+ enable_tool_shortlisting: bool = False
+ max_selected_tools: int = 30
+
+ runner: RunnerName | None = None
+ model_settings: ModelSettings | None = None
+
+ @classmethod
+ def _get_instance_class(cls):
+ from .autoskill_instance import AutoSkillAgentInstance
+ return AutoSkillAgentInstance
+
+ @classmethod
+ def _get_instance_class_ref(cls) -> str:
+ return "exgentic.agents.autoskill.autoskill_instance:AutoSkillAgentInstance"
+
+ def _get_instance_kwargs(self, session_id: str) -> dict[str, Any]:
+ return {
+ "session_id": session_id,
+ "model": self.model,
+ "skill_model": self.skill_model or self.model,
+ "retrieve_k": self.retrieve_k,
+ "retrieval_threshold": self.retrieval_threshold,
+ "bm25_weight": self.bm25_weight,
+ "dedupe_similarity_threshold": self.dedupe_similarity_threshold,
+ "embedding_model": self.embedding_model,
+ "enable_query_rewrite": self.enable_query_rewrite,
+ "max_context_chars": self.max_context_chars,
+ "shuffle_mode": self.shuffle_mode,
+ "model_settings": self.model_settings,
+ "benchmark_id": self.benchmark_id,
+ "enable_tool_shortlisting": self.enable_tool_shortlisting,
+ "max_selected_tools": self.max_selected_tools,
+ }
+
+ @property
+ def model_name(self) -> str:
+ return str(self.model).split("/")[-1]
+
+ def get_models_names(self) -> list[str]:
+ names = [str(self.model)]
+ sm = self.skill_model or self.model
+ if sm != self.model:
+ names.append(str(sm))
+ return names
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/autoskill_instance.py b/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/autoskill_instance.py
new file mode 100644
index 00000000..b2d9bff2
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/autoskill_instance.py
@@ -0,0 +1,636 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import json
+import time
+from datetime import datetime
+from typing import Any, Dict, List, Optional, Union
+
+import litellm
+from litellm import (
+ ChatCompletionAssistantMessage,
+ ChatCompletionSystemMessage,
+ ChatCompletionToolMessage,
+ ChatCompletionUserMessage,
+)
+
+from ...core.agent_instance import AgentInstance
+from ...core.types import (
+ Action,
+ ActionType,
+ Message,
+ MessageAction,
+ MessageObservation,
+ MessagePayload,
+ ModelSettings,
+ Observation,
+)
+from ...utils.cost import LiteLLMCostReport
+from ...utils.settings import get_settings
+
+from .prompts import QUERY_REWRITE_PROMPT, SKILL_CONTEXT_TEMPLATE, SKILL_ENTRY_TEMPLATE
+from .skill_extraction import extract_skills_from_trace
+from .skill_maintenance import maintain_skill
+from .skill_retrieval import compute_embedding, hybrid_search
+from .skill_store import SkillStore
+
+try:
+ from ...agents.litellm_tool_calling.utils import ToolCall, ToolsActionsRegistry
+except ImportError:
+ ToolsActionsRegistry = None
+ ToolCall = dict
+
+from ..tool_shortlisting import shortlist_tools
+
+settings = get_settings()
+
+
+class AutoSkillAgentInstance(AgentInstance):
+
+ def __init__(
+ self,
+ session_id: str,
+ model: str = "gpt-4o",
+ skill_model: str = "gpt-4o",
+ retrieve_k: int = 5,
+ retrieval_threshold: float = 0.3,
+ bm25_weight: float = 0.1,
+ dedupe_similarity_threshold: float = 0.4,
+ embedding_model: str = "text-embedding-3-small",
+ enable_query_rewrite: bool = True,
+ max_context_chars: int = 6000,
+ shuffle_mode: str = "isolated",
+ model_settings: Optional[ModelSettings] = None,
+ benchmark_id: Optional[str] = None,
+ enable_tool_shortlisting: bool = False,
+ max_selected_tools: int = 30,
+ ) -> None:
+ super().__init__(session_id)
+
+ self.model = model
+ self.skill_model = skill_model
+ self.retrieve_k = retrieve_k
+ self.retrieval_threshold = retrieval_threshold
+ self.bm25_weight = bm25_weight
+ self.dedupe_similarity_threshold = dedupe_similarity_threshold
+ self.embedding_model = embedding_model
+ self.enable_query_rewrite = enable_query_rewrite
+ self.max_context_chars = max_context_chars
+ self.shuffle_mode = shuffle_mode
+ self.benchmark_id = benchmark_id
+ self.enable_tool_shortlisting = enable_tool_shortlisting
+ self.max_selected_tools = max_selected_tools
+
+ if model_settings is None:
+ self._model_settings = ModelSettings()
+ elif isinstance(model_settings, ModelSettings):
+ self._model_settings = model_settings
+ else:
+ self._model_settings = ModelSettings()
+
+ self._cost = LiteLLMCostReport.initialize_empty(model_name=self.model)
+ self._store: Optional[SkillStore] = None
+
+ self.messages: list[
+ Union[
+ ChatCompletionAssistantMessage,
+ ChatCompletionToolMessage,
+ ChatCompletionUserMessage,
+ ChatCompletionSystemMessage,
+ ]
+ ] = []
+ self._step_count: int = 0
+
+ self._registry: Optional[ToolsActionsRegistry] = None
+ self._all_actions: list[ActionType] = []
+
+ self._observation_log: List[Dict[str, Any]] = []
+ self._action_log: List[Dict[str, Any]] = []
+
+ def _log_failure(
+ self, component: str, error: Exception, context: Dict[str, Any]
+ ) -> None:
+ try:
+ log_path = self.paths.agent_dir / "autoskill_failures.jsonl"
+ log_path.parent.mkdir(parents=True, exist_ok=True)
+ entry = {
+ "timestamp": datetime.now().isoformat(),
+ "session_id": self.session_id,
+ "component": component,
+ "error_type": type(error).__name__,
+ "error_message": str(error)[:2000],
+ **{k: str(v)[:2000] if isinstance(v, str) else v
+ for k, v in context.items()},
+ }
+ with open(log_path, "a", encoding="utf-8") as fh:
+ fh.write(json.dumps(entry, ensure_ascii=False) + "\n")
+ except Exception:
+ pass
+
+ def start(
+ self,
+ task: str,
+ context: Dict[str, Any],
+ actions: list[ActionType],
+ ) -> None:
+ super().start(task, context, actions)
+
+ self._all_actions = list(self.actions)
+ if ToolsActionsRegistry is not None:
+ self._registry = ToolsActionsRegistry(self._all_actions)
+
+ task_group = str(
+ context.get("task_group")
+ or context.get("task_id")
+ or context.get("task_name")
+ or "default"
+ )
+ self._store = SkillStore.get_or_create(
+ shuffle_mode=self.shuffle_mode,
+ task_group=task_group,
+ benchmark_id=self.benchmark_id,
+ )
+ self._store.increment_session()
+
+ skill_context = self._retrieve_skills(task, context)
+
+ system_content = self._build_system_prompt(skill_context)
+ self._add_message(
+ ChatCompletionSystemMessage(role="system", content=system_content)
+ )
+
+ content_parts: list[Any] = []
+ ctx = ""
+ if self.context:
+ for k, v in self.context.items():
+ if isinstance(v, dict) and v.get("type") == "image_url":
+ content_parts.append({"type": "image_url", "image_url": {"url": v["data"], "detail": "high"}})
+ else:
+ ctx += f"\n<{k}>\n{v}\n"
+
+ text_content = f"{self.task}\n{ctx}"
+ if content_parts:
+ content_parts.insert(0, {"type": "text", "text": text_content})
+ self._add_message(ChatCompletionUserMessage(role="user", content=content_parts))
+ else:
+ self._add_message(
+ ChatCompletionUserMessage(role="user", content=text_content)
+ )
+
+ self.logger.info(
+ "AutoSkill instance started store=%s session_count=%d "
+ "skill_count=%d benchmark=%s tools=%d",
+ self._store.store_id,
+ self._store.session_count,
+ self._store.skill_count,
+ self.benchmark_id or "(none)",
+ len(self._all_actions),
+ )
+
+ def react(self, observation: Optional[Observation]) -> Optional[Action]:
+ self._step_count += 1
+
+ self._observe(observation)
+ self._log_observation(observation)
+
+ tools = self._assistant_tools()
+ response = self._completion(
+ model=self.model,
+ messages=self.messages,
+ tools=tools if tools else None,
+ )
+
+ if response is None:
+ self.logger.error("AutoSkill: LLM returned None response")
+ return None
+
+ if response.usage:
+ self._cost.update_cost_from_tokens(
+ response.usage.prompt_tokens,
+ response.usage.completion_tokens,
+ )
+
+ choice = response["choices"][0]
+ message = choice["message"]
+ finish_reason = choice.get("finish_reason")
+
+ if finish_reason == "tool_calls" and self._registry is not None:
+ tool_calls = self._extract_tool_calls(message)
+ self._add_message(
+ ChatCompletionAssistantMessage(
+ role="assistant",
+ tool_calls=[
+ {
+ "id": tc["id"],
+ "type": "function",
+ "function": {
+ "name": tc["name"],
+ "arguments": tc["arguments"],
+ },
+ }
+ for tc in tool_calls
+ ],
+ )
+ )
+ actions = self._registry.tool_calls_to_action(tool_calls)
+
+ for tc in tool_calls:
+ self._action_log.append({
+ "step": self._step_count,
+ "action": tc["name"],
+ "arguments": tc["arguments"],
+ })
+
+ self.logger.info("AutoSkill step %d: tool_calls=%s", self._step_count,
+ [tc["name"] for tc in tool_calls])
+ return actions
+ else:
+ content = message.content if message.content else ""
+ self._add_message(
+ ChatCompletionAssistantMessage(
+ role="assistant", content=content
+ )
+ )
+ self._action_log.append({
+ "step": self._step_count,
+ "action": "message",
+ "content": content,
+ })
+ self.logger.info("AutoSkill step %d: message response", self._step_count)
+ return MessageAction(arguments=Message(content=content))
+
+ def close(self) -> None:
+ store = self._store
+ if store is None:
+ return
+
+ action_taken = "no_extraction"
+ skill_name = ""
+
+ if self._observation_log or self._action_log:
+ try:
+ session_trace = self._build_session_trace()
+ candidates = extract_skills_from_trace(
+ task=self.task if self.task else "",
+ benchmark_id=self.benchmark_id or "",
+ session_trace=session_trace,
+ llm_call=self._llm_call_simple,
+ model=self.skill_model,
+ logger=self.logger,
+ )
+
+ if candidates:
+ candidate = candidates[0]
+ action_taken, _ = maintain_skill(
+ candidate=candidate,
+ store=store,
+ llm_call=self._llm_call_simple,
+ model=self.skill_model,
+ embedding_model=self.embedding_model,
+ bm25_weight=self.bm25_weight,
+ dedupe_similarity_threshold=self.dedupe_similarity_threshold,
+ logger=self.logger,
+ )
+ skill_name = candidate.name
+ self.logger.info(
+ "AutoSkill close: action=%s skill=%s",
+ action_taken, skill_name,
+ )
+
+ except Exception as exc:
+ self.logger.warning("AutoSkill: skill extraction/maintenance failed: %s", exc)
+ self._log_failure("skill_evolution", exc, {
+ "observation_count": len(self._observation_log),
+ "action_count": len(self._action_log),
+ })
+
+ store.record_learning(
+ session_id=self.session_id,
+ task_id=str(self.context.get("task_id", "") if self.context else ""),
+ benchmark_id=self.benchmark_id or "",
+ action=action_taken,
+ skill_name=skill_name,
+ )
+
+ try:
+ cp = str(self.paths.agent_dir / "skillstore_checkpoint.json")
+ store.save_checkpoint(cp)
+ txt = str(self.paths.agent_dir / "skillbank.txt")
+ store.save_skills_text(txt)
+ except Exception as exc:
+ self.logger.warning("AutoSkill: failed to save checkpoint: %s", exc)
+
+ def get_cost(self) -> LiteLLMCostReport:
+ return self._cost
+
+ def _retrieve_skills(self, task: str, context: Dict[str, Any]) -> str:
+ store = self._store
+ if store is None or store.skill_count == 0:
+ return ""
+
+ query = task
+ if self.enable_query_rewrite and task:
+ try:
+ ctx_parts = []
+ for _, v in context.items():
+ if isinstance(v, str):
+ ctx_parts.append(v)
+ ctx_str = " ".join(ctx_parts)
+ rewritten = self._llm_call_simple(
+ self.skill_model,
+ QUERY_REWRITE_PROMPT.format(task=task, context=ctx_str),
+ )
+ if rewritten and len(rewritten.strip()) > 5:
+ query = rewritten.strip()
+ self.logger.info("AutoSkill retrieval: query rewritten to '%s'", query[:100])
+ except Exception as exc:
+ self.logger.debug("AutoSkill retrieval: query rewrite failed: %s", exc)
+
+ query_embedding = None
+ try:
+ query_embedding = compute_embedding(query, model=self.embedding_model)
+ except Exception as exc:
+ self.logger.warning("AutoSkill retrieval: failed to compute query embedding: %s", exc)
+
+ results = hybrid_search(
+ store=store,
+ query=query,
+ query_embedding=query_embedding,
+ top_k=self.retrieve_k,
+ threshold=self.retrieval_threshold,
+ bm25_weight=self.bm25_weight,
+ embedding_model=self.embedding_model,
+ )
+
+ if not results:
+ self.logger.info("AutoSkill retrieval: no skills above threshold %.2f", self.retrieval_threshold)
+ return ""
+
+ self.logger.info(
+ "AutoSkill retrieval: %d skills retrieved (top score=%.3f)",
+ len(results), results[0][1],
+ )
+
+ skills_block = ""
+ char_budget = self.max_context_chars
+ for skill, _ in results:
+ entry_text = SKILL_ENTRY_TEMPLATE.format(
+ name=skill.name,
+ description=skill.description,
+ tags=", ".join(skill.tags),
+ triggers=", ".join(skill.triggers),
+ instructions=skill.instructions,
+ )
+ if len(skills_block) + len(entry_text) > char_budget:
+ break
+ skills_block += entry_text + "\n"
+
+ return SKILL_CONTEXT_TEMPLATE.format(skills_block=skills_block)
+
+ def _build_system_prompt(self, skill_context: str) -> str:
+ parts = [
+ "You are an expert agent that completes tasks using available tools.",
+ "Think step-by-step before acting.",
+ "Use available tools to interact with the environment.",
+ "When you are confident in your solution, use the finish/submit tool.",
+ ]
+
+ if skill_context:
+ parts.extend(["", skill_context])
+ else:
+ parts.extend([
+ "",
+ "## Skills",
+ "(No accumulated skills yet. This is an early session.)",
+ ])
+
+ return "\n".join(parts)
+
+ def _add_message(self, message: Any) -> None:
+ self.messages.append(message)
+
+ def _observe(self, observation: Optional[Observation]) -> None:
+ if observation is None:
+ return
+
+ observations = observation.to_observation_list()
+ if observation.is_empty():
+ if not any(obs.invoking_actions for obs in observations):
+ return
+
+ for obs in observations:
+ if isinstance(obs, MessageObservation) and isinstance(
+ obs.result, MessagePayload
+ ):
+ self._add_message(
+ ChatCompletionUserMessage(
+ role="user", content=obs.result.message
+ )
+ )
+ continue
+
+ if len(obs.invoking_actions) > 0:
+ invoking = obs.invoking_actions[0]
+ if invoking.name == "message":
+ self._add_message(
+ ChatCompletionUserMessage(
+ role="user", content=str(obs)
+ )
+ )
+ continue
+
+ action_id = invoking.id
+ tool_call_id = invoking.id
+ if not (
+ isinstance(tool_call_id, str)
+ and tool_call_id.startswith("call_")
+ ):
+ if self._registry is not None:
+ tool_call_id = (
+ self._registry.action_id_to_tool_call_id.get(
+ action_id, tool_call_id
+ )
+ )
+
+ value = obs.result
+ try:
+ content = json.dumps(
+ value, ensure_ascii=False, separators=(",", ":")
+ )
+ except TypeError:
+ content = str(value)
+
+ if tool_call_id is not None:
+ self._add_message(
+ ChatCompletionToolMessage(
+ role="tool",
+ tool_call_id=tool_call_id,
+ content=content,
+ )
+ )
+ else:
+ self._add_message(
+ ChatCompletionUserMessage(
+ role="user",
+ content=f"Tool result: {content}",
+ )
+ )
+ else:
+ self._add_message(
+ ChatCompletionUserMessage(
+ role="user", content=str(obs)
+ )
+ )
+
+ def _log_observation(self, observation: Optional[Observation]) -> None:
+ if observation is None or observation.is_empty():
+ return
+
+ for obs in observation.to_observation_list():
+ result = obs.result
+ if result is None:
+ continue
+
+ entry: Dict[str, Any] = {"step": self._step_count}
+ if isinstance(result, str):
+ entry["content"] = result
+ elif isinstance(result, dict):
+ entry["content"] = json.dumps(result, ensure_ascii=False)
+ else:
+ entry["content"] = str(result)
+
+ if obs.invoking_actions:
+ entry["action"] = obs.invoking_actions[0].name
+
+ self._observation_log.append(entry)
+
+ def _assistant_tools(self) -> list[dict[str, Any]]:
+ if self._registry is None:
+ return []
+ tools = self._registry.openai_tools()
+ if not self.enable_tool_shortlisting:
+ return tools
+
+ def _cost_cb(usage):
+ if usage:
+ self._cost.update_cost_from_tokens(
+ usage.prompt_tokens, usage.completion_tokens
+ )
+
+ return shortlist_tools(
+ tools=tools,
+ max_selected=self.max_selected_tools,
+ messages=self.messages,
+ completion_fn=self._completion,
+ model=self.model,
+ logger=self.logger,
+ cost_callback=_cost_cb,
+ )
+
+ @staticmethod
+ def _extract_tool_calls(message: Any) -> list[dict[str, str]]:
+ if not hasattr(message, "tool_calls") or not message.tool_calls:
+ return []
+ tool_calls = []
+ for tc in message.tool_calls:
+ tool_calls.append({
+ "name": tc.function.name,
+ "arguments": tc.function.arguments,
+ "id": tc.id,
+ })
+ return tool_calls
+
+ def _completion(self, **kwargs) -> Any:
+ call_kwargs = self._model_settings.model_dump(
+ exclude_none=True,
+ exclude={"num_retries", "retry_after", "retry_strategy"},
+ )
+ call_kwargs.update(kwargs)
+ if call_kwargs.get("tools") is None:
+ call_kwargs.pop("tools", None)
+
+ max_attempts = 3
+ for attempt in range(max_attempts):
+ try:
+ response = litellm.completion(**call_kwargs)
+ return response
+ except Exception as exc:
+ self.logger.warning(
+ "AutoSkill LLM call attempt %d/%d failed: %s",
+ attempt + 1, max_attempts, exc,
+ )
+ if attempt + 1 >= max_attempts:
+ raise
+ time.sleep(2 ** attempt)
+ return None
+
+ def _llm_call_simple(
+ self,
+ model: str,
+ prompt: str,
+ *,
+ json_mode: bool = False,
+ ) -> str:
+ kwargs: Dict[str, Any] = {
+ "model": model,
+ "messages": [{"role": "user", "content": prompt}],
+ "temperature": 0.0,
+ }
+ if json_mode:
+ kwargs["response_format"] = {"type": "json_object"}
+
+ max_attempts = 3
+ for attempt in range(max_attempts):
+ try:
+ resp = litellm.completion(**kwargs)
+ if resp.usage:
+ self._cost.update_cost_from_tokens(
+ resp.usage.prompt_tokens,
+ resp.usage.completion_tokens,
+ )
+ content = resp.choices[0].message.content
+ if content is None:
+ raise ValueError("LLM returned None content")
+ return content
+ except Exception as exc:
+ self.logger.warning(
+ "AutoSkill simple LLM call attempt %d/%d failed: %s",
+ attempt + 1, max_attempts, exc,
+ )
+ if attempt + 1 >= max_attempts:
+ self._log_failure("llm_call", exc, {
+ "model": model,
+ "prompt_length": len(prompt),
+ "attempts": max_attempts,
+ })
+ raise
+ time.sleep(2 ** attempt)
+ return ""
+
+ def _build_session_trace(self) -> str:
+ events: List[Dict[str, Any]] = []
+ for entry in self._action_log:
+ events.append({"type": "action", **entry})
+ for entry in self._observation_log:
+ events.append({"type": "observation", **entry})
+
+ events.sort(key=lambda e: (e.get("step", 0), 0 if e["type"] == "action" else 1))
+
+ lines: List[str] = []
+ for event in events:
+ step = event.get("step", "?")
+ if event["type"] == "action":
+ action = event.get("action", "?")
+ args = event.get("arguments", event.get("content", ""))
+ lines.append(f"[Step {step}] Action: {action}")
+ if args:
+ lines.append(f" Args: {str(args)}")
+ else:
+ action = event.get("action", "env")
+ content = event.get("content", "")
+ lines.append(f"[Step {step}] Observation from {action}:")
+ lines.append(f" {content}")
+
+ return "\n".join(lines) if lines else "(No session trace recorded)"
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/prompts.py b/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/prompts.py
new file mode 100644
index 00000000..eebe79fd
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/prompts.py
@@ -0,0 +1,153 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+QUERY_REWRITE_PROMPT = """\
+You are a retrieval query rewriter. Your job is to rewrite the current user task \
+into a concise, standalone search query for skill retrieval.
+
+Core rules:
+- Produce exactly ONE line of output: the rewritten query.
+- Resolve references ("it", "this", "the above") using the provided context.
+- Keep only retrieval-relevant constraints (format, audience, quality, domain).
+- Preserve the task anchor (what the task is about).
+- Do NOT include generic process words without a concrete topic anchor.
+
+Task: {task}
+Context: {context}
+
+Rewritten query:"""
+
+
+SKILL_EXTRACTION_PROMPT = """\
+You are a skill extractor that turns agent interaction traces into reusable skills.
+
+## Extraction Principles
+- Treat the task description and environment observations as primary evidence.
+- Extract ONLY when there are durable, reusable constraints, policies, workflows, \
+or strategies that would help in FUTURE similar tasks.
+- Do NOT extract one-shot task-specific facts or generic "be helpful" patterns.
+- Capture HOW TO DO similar tasks, rather than this-instance facts.
+- Remove case-specific entities (names, URLs, dates) and preserve only portable rules.
+- Do NOT invent workflow steps unless explicitly demonstrated in the trace.
+- If nothing reusable is found, return an empty skills list.
+
+## Session Information
+Task: {task}
+
+## Session Trace (Actions & Observations)
+{session_trace}
+
+## Output Format
+Return a JSON object with this schema:
+{{
+ "skills": [
+ {{
+ "name": "",
+ "description": "",
+ "instructions": "",
+ "triggers": ["", "", ...],
+ "tags": ["", "", ...],
+ "confidence":
+ }}
+ ]
+}}
+
+If nothing reusable is detected, return: {{"skills": []}}
+"""
+
+SKILL_JUDGE_PROMPT = """\
+You are a skill set manager. Given a newly extracted skill candidate and the most \
+similar existing skill from the skill bank, decide the appropriate action.
+
+## Decision Procedure
+1. Check if the candidate represents the same capability as the existing skill \
+(same job-to-be-done, same deliverable type, overlapping constraints).
+2. Apply discard gate: reject generic, low-signal, non-portable candidates.
+3. Compare on four axes: job-to-be-done, deliverable type, hard constraints/success \
+criteria, and required tools/workflow.
+4. Choose "merge" ONLY when they are the same capability after removing instance details.
+5. Choose "add" when the candidate is a distinct durable capability.
+6. Choose "discard" when the candidate is too generic or non-reusable.
+
+## Candidate Skill
+Name: {candidate_name}
+Description: {candidate_description}
+Instructions: {candidate_instructions}
+Triggers: {candidate_triggers}
+Tags: {candidate_tags}
+
+## Most Similar Existing Skill (may be empty if no skills exist)
+Name: {existing_name}
+Description: {existing_description}
+Instructions: {existing_instructions}
+Triggers: {existing_triggers}
+Tags: {existing_tags}
+Similarity Score: {similarity_score}
+
+## Output Format
+Return a JSON object:
+{{
+ "action": "add" | "merge" | "discard",
+ "target_skill_id": "",
+ "reason": ""
+}}
+"""
+
+SKILL_MERGE_PROMPT = """\
+You are a skill merger. Combine an existing skill with a new candidate into one \
+improved skill that preserves the best of both.
+
+## Merge Rules
+- Preserve the original capability identity (name and core goal).
+- Perform semantic union rather than raw concatenation.
+- Import only reusable, non-conflicting additions from the candidate.
+- Avoid regressions: keep important checks from the existing skill.
+- Remove case-specific entities and one-off facts.
+- Do NOT invent any new standards or details not present in either skill.
+- Deduplicate sections, bullets, triggers, tags.
+- Keep language consistent across all fields.
+
+## Existing Skill
+Name: {existing_name}
+Description: {existing_description}
+Instructions: {existing_instructions}
+Triggers: {existing_triggers}
+Tags: {existing_tags}
+
+## Candidate Skill (new evidence)
+Name: {candidate_name}
+Description: {candidate_description}
+Instructions: {candidate_instructions}
+Triggers: {candidate_triggers}
+Tags: {candidate_tags}
+
+## Output Format
+Return a JSON object with the merged skill:
+{{
+ "name": "",
+ "description": "",
+ "instructions": "",
+ "triggers": ["", ...],
+ "tags": ["", ...]
+}}
+"""
+
+SKILL_CONTEXT_TEMPLATE = """\
+## Retrieved Skills (from accumulated experience)
+The following skills were retrieved based on relevance to the current task. \
+Use a skill ONLY when it directly matches the current intent. \
+Otherwise, ignore all retrieved skills and act normally. \
+Never explicitly mention that skills were retrieved/injected.
+
+{skills_block}
+"""
+
+SKILL_ENTRY_TEMPLATE = """\
+### Skill: {name}
+- **Description**: {description}
+- **Tags**: {tags}
+- **Triggers**: {triggers}
+
+**Instructions**:
+{instructions}
+"""
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/skill_extraction.py b/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/skill_extraction.py
new file mode 100644
index 00000000..4478b4b4
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/skill_extraction.py
@@ -0,0 +1,141 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import json
+import re
+import uuid
+from typing import Any, Callable, Dict, List, Optional
+
+from .prompts import SKILL_EXTRACTION_PROMPT
+from .skill_store import SkillEntry
+
+
+def extract_json_from_text(text: str) -> Optional[Dict[str, Any]]:
+ text = text.strip()
+ try:
+ return json.loads(text)
+ except (json.JSONDecodeError, ValueError):
+ pass
+
+ match = re.search(r"```(?:json)?\s*\n?(.*?)\n?```", text, re.DOTALL)
+ if match:
+ try:
+ return json.loads(match.group(1).strip())
+ except (json.JSONDecodeError, ValueError):
+ pass
+
+ start = text.find("{")
+ if start >= 0:
+ depth = 0
+ for i in range(start, len(text)):
+ if text[i] == "{":
+ depth += 1
+ elif text[i] == "}":
+ depth -= 1
+ if depth == 0:
+ try:
+ return json.loads(text[start:i + 1])
+ except (json.JSONDecodeError, ValueError):
+ break
+ return None
+
+
+def _repair_json_via_llm(
+ raw_text: str,
+ llm_call: Callable[[str, str], str],
+ model: str,
+) -> Optional[Dict[str, Any]]:
+ repair_prompt = (
+ "The following text was supposed to be valid JSON matching the schema "
+ '{"skills": [{"name": str, "description": str, "instructions": str, '
+ '"triggers": [str], "tags": [str], "confidence": float}]} '
+ "but it is malformed. Please fix it and return ONLY valid JSON. "
+ 'If extraction fails, output {"skills": []}.\n\n'
+ f"Malformed text:\n{raw_text[:4000]}"
+ )
+ try:
+ repaired = llm_call(model, repair_prompt)
+ if repaired:
+ return extract_json_from_text(repaired)
+ except Exception:
+ pass
+ return None
+
+
+def extract_skills_from_trace(
+ task: str,
+ benchmark_id: str,
+ session_trace: str,
+ llm_call: Callable[[str, str], str],
+ model: str,
+ logger: Optional[Any] = None,
+) -> List[SkillEntry]:
+ import logging
+ _logger = logger or logging.getLogger(__name__)
+
+ prompt = SKILL_EXTRACTION_PROMPT.format(
+ task=task,
+ session_trace=session_trace,
+ )
+
+ _logger.info(
+ "AutoSkill extraction: task='%s', trace_len=%d chars",
+ task[:80], len(session_trace),
+ )
+
+ raw = llm_call(model, prompt)
+ if not raw:
+ _logger.info("AutoSkill extraction: LLM returned empty response")
+ return []
+
+ _logger.debug("AutoSkill extraction: raw LLM response length=%d", len(raw))
+
+ parsed = extract_json_from_text(raw)
+ if parsed is None:
+ _logger.info("AutoSkill extraction: levels 1-3 JSON parse failed, attempting LLM repair")
+ parsed = _repair_json_via_llm(raw, llm_call, model)
+ if not parsed or not isinstance(parsed, dict):
+ _logger.warning("AutoSkill extraction: all 4 JSON recovery levels failed")
+ return []
+
+ skills_data = parsed.get("skills", [])
+ if not isinstance(skills_data, list):
+ _logger.warning("AutoSkill extraction: 'skills' field is not a list")
+ return []
+
+ _logger.info("AutoSkill extraction: LLM returned %d skill candidates", len(skills_data))
+
+ results: List[SkillEntry] = []
+ for item in skills_data:
+ if not isinstance(item, dict):
+ continue
+
+ name = item.get("name", "").strip()
+ description = item.get("description", "").strip()
+ instructions = item.get("instructions", "").strip()
+ confidence = float(item.get("confidence", 0.6))
+
+ if not name or not description:
+ _logger.debug("AutoSkill extraction: skipping candidate with empty name/description")
+ continue
+ if not instructions:
+ instructions = description
+
+ entry = SkillEntry(
+ id=str(uuid.uuid4()),
+ name=name,
+ description=description,
+ instructions=instructions,
+ triggers=item.get("triggers", [])[:8],
+ tags=item.get("tags", [])[:8],
+ confidence=confidence,
+ )
+ _logger.info(
+ "AutoSkill extraction: extracted skill '%s' (confidence=%.2f)",
+ name, confidence,
+ )
+ results.append(entry)
+
+ return results
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/skill_maintenance.py b/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/skill_maintenance.py
new file mode 100644
index 00000000..fdc1ecd1
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/skill_maintenance.py
@@ -0,0 +1,207 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import json
+from typing import Any, Callable, Optional, Tuple
+
+from .prompts import SKILL_JUDGE_PROMPT, SKILL_MERGE_PROMPT
+from .skill_extraction import extract_json_from_text
+from .skill_retrieval import hybrid_search, compute_embedding
+from .skill_store import SkillEntry, SkillStore
+
+
+def judge_skill(
+ candidate: SkillEntry,
+ existing: Optional[SkillEntry],
+ similarity_score: float,
+ llm_call: Callable[[str, str], str],
+ model: str,
+) -> Tuple[str, Optional[str], str]:
+ prompt = SKILL_JUDGE_PROMPT.format(
+ candidate_name=candidate.name,
+ candidate_description=candidate.description,
+ candidate_instructions=candidate.instructions[:2000],
+ candidate_triggers=", ".join(candidate.triggers),
+ candidate_tags=", ".join(candidate.tags),
+ existing_name=existing.name if existing else "(none - no existing skill)",
+ existing_description=existing.description if existing else "",
+ existing_instructions=(existing.instructions[:2000] if existing else ""),
+ existing_triggers=", ".join(existing.triggers) if existing else "",
+ existing_tags=", ".join(existing.tags) if existing else "",
+ similarity_score=f"{similarity_score:.3f}",
+ )
+
+ raw = llm_call(model, prompt)
+ parsed = extract_json_from_text(raw)
+
+ if parsed and isinstance(parsed, dict):
+ action = parsed.get("action", "discard").lower().strip()
+ target_id = parsed.get("target_skill_id")
+ reason = parsed.get("reason", "")
+ if action in ("add", "merge", "discard"):
+ if action == "merge" and existing:
+ return action, existing.id, reason
+ elif action == "merge" and not existing:
+ return "add", None, reason + " (no target for merge, adding instead)"
+ return action, target_id, reason
+
+ if existing and similarity_score >= 0.82:
+ return "merge", existing.id, "high similarity (deterministic fallback)"
+ elif not existing or similarity_score <= 0.22:
+ return "add", None, "low similarity, distinct skill (deterministic fallback)"
+ elif similarity_score >= 0.50:
+ return "merge", existing.id, "moderate similarity (deterministic fallback)"
+ else:
+ return "add", None, "below merge threshold (deterministic fallback)"
+
+
+def merge_skills(
+ existing: SkillEntry,
+ candidate: SkillEntry,
+ llm_call: Callable[[str, str], str],
+ model: str,
+) -> SkillEntry:
+ prompt = SKILL_MERGE_PROMPT.format(
+ existing_name=existing.name,
+ existing_description=existing.description,
+ existing_instructions=existing.instructions[:3000],
+ existing_triggers=json.dumps(existing.triggers, ensure_ascii=False),
+ existing_tags=json.dumps(existing.tags, ensure_ascii=False),
+ candidate_name=candidate.name,
+ candidate_description=candidate.description,
+ candidate_instructions=candidate.instructions[:3000],
+ candidate_triggers=json.dumps(candidate.triggers, ensure_ascii=False),
+ candidate_tags=json.dumps(candidate.tags, ensure_ascii=False),
+ )
+
+ raw = llm_call(model, prompt)
+ parsed = extract_json_from_text(raw)
+
+ merged = SkillEntry(
+ id=existing.id,
+ name=existing.name,
+ description=existing.description,
+ instructions=existing.instructions,
+ triggers=list(existing.triggers),
+ tags=list(existing.tags),
+ examples=list(existing.examples),
+ version=existing.version,
+ confidence=max(existing.confidence, candidate.confidence),
+ created_at=existing.created_at,
+ updated_at=existing.updated_at,
+ )
+
+ if parsed and isinstance(parsed, dict):
+ if parsed.get("name"):
+ merged.name = parsed["name"]
+ if parsed.get("description"):
+ merged.description = parsed["description"]
+ if parsed.get("instructions"):
+ merged.instructions = parsed["instructions"]
+ if parsed.get("triggers"):
+ merged.triggers = list(set(existing.triggers + parsed["triggers"]))[:10]
+ if parsed.get("tags"):
+ merged.tags = list(set(existing.tags + parsed["tags"]))[:10]
+ else:
+ merged.triggers = list(set(existing.triggers + candidate.triggers))[:10]
+ merged.tags = list(set(existing.tags + candidate.tags))[:10]
+ if candidate.instructions and candidate.instructions not in existing.instructions:
+ merged.instructions = (
+ existing.instructions + "\n\n## Updated Constraints\n" + candidate.instructions
+ )
+
+ merged.bump_version()
+ return merged
+
+
+def maintain_skill(
+ candidate: SkillEntry,
+ store: SkillStore,
+ llm_call: Callable[[str, str], str],
+ model: str,
+ embedding_model: str = "text-embedding-3-small",
+ bm25_weight: float = 0.1,
+ dedupe_similarity_threshold: float = 0.4,
+ logger: Any = None,
+) -> Tuple[str, Optional[SkillEntry]]:
+ import logging
+ _logger = logger or logging.getLogger(__name__)
+
+ # Compute candidate embedding for retrieval
+ candidate_text = candidate.to_search_text()
+ candidate_embedding = None
+ try:
+ candidate_embedding = compute_embedding(candidate_text, model=embedding_model)
+ except Exception as exc:
+ _logger.warning(
+ "AutoSkill maintenance: failed to compute candidate embedding for '%s': %s",
+ candidate.name, exc,
+ )
+ _logger.info(
+ "AutoSkill maintenance: candidate='%s', embedding_dim=%d",
+ candidate.name, len(candidate_embedding) if candidate_embedding else 0,
+ )
+
+ results = hybrid_search(
+ store=store,
+ query=candidate_text,
+ query_embedding=candidate_embedding,
+ top_k=1,
+ threshold=0.0,
+ bm25_weight=bm25_weight,
+ embedding_model=embedding_model,
+ )
+
+ existing: Optional[SkillEntry] = None
+ similarity_score = 0.0
+ if results:
+ existing, similarity_score = results[0]
+ _logger.info(
+ "AutoSkill maintenance: best match='%s' (score=%.3f)",
+ existing.name, similarity_score,
+ )
+ else:
+ _logger.info("AutoSkill maintenance: no existing skills in store")
+
+ if existing and similarity_score < dedupe_similarity_threshold:
+ _logger.info(
+ "AutoSkill maintenance: similarity %.3f < threshold %.3f, skipping merge consideration",
+ similarity_score, dedupe_similarity_threshold,
+ )
+ existing = None
+ similarity_score = 0.0
+
+ action, _, reason = judge_skill(
+ candidate, existing, similarity_score, llm_call, model,
+ )
+ _logger.info(
+ "AutoSkill maintenance: judge decision='%s', reason='%s'",
+ action, reason[:100],
+ )
+
+ if action == "discard":
+ _logger.info("AutoSkill maintenance: discarded candidate '%s'", candidate.name)
+ return "discard", None
+ elif action == "merge" and existing:
+ merged = merge_skills(existing, candidate, llm_call, model)
+ store.update_skill(merged)
+ if candidate_embedding:
+ new_emb = compute_embedding(merged.to_search_text(), model=embedding_model)
+ if new_emb:
+ store.set_embedding(merged.id, new_emb)
+ _logger.info(
+ "AutoSkill maintenance: merged into '%s' (v%s → v%s)",
+ merged.name, existing.version, merged.version,
+ )
+ return "merge", merged
+ else:
+ store.add_skill(candidate)
+ if candidate_embedding:
+ store.set_embedding(candidate.id, candidate_embedding)
+ _logger.info(
+ "AutoSkill maintenance: added new skill '%s' (id=%s)",
+ candidate.name, candidate.id,
+ )
+ return "add", candidate
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/skill_retrieval.py b/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/skill_retrieval.py
new file mode 100644
index 00000000..c4d24c41
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/skill_retrieval.py
@@ -0,0 +1,159 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import logging
+import math
+import re
+from collections import Counter
+from typing import Dict, List, Optional, Tuple
+
+from .skill_store import SkillEntry, SkillStore
+
+logger = logging.getLogger(__name__)
+
+_TOKEN_RE = re.compile(r"[A-Za-z0-9_]+|[\u4e00-\u9fff]|[^\W\d_]+", re.UNICODE)
+_STOPWORDS = frozenset([
+ "the", "a", "an", "is", "are", "was", "were", "be", "been",
+ "being", "have", "has", "had", "do", "does", "did", "will",
+ "would", "could", "should", "may", "might", "shall", "can",
+ "to", "of", "in", "for", "on", "with", "at", "by", "from",
+ "and", "or", "but", "not", "if", "then", "else", "when",
+ "that", "this", "it", "its", "as", "so", "no", "yes",
+])
+
+
+def tokenize(text: str) -> List[str]:
+ tokens = _TOKEN_RE.findall(text.lower())
+ return [t for t in tokens if t not in _STOPWORDS and len(t) > 1]
+
+
+_st_model = None
+_st_model_name = None
+
+
+def _get_st_model(model_name: str = "all-MiniLM-L6-v2"):
+ global _st_model, _st_model_name
+ if _st_model is None or _st_model_name != model_name:
+ from sentence_transformers import SentenceTransformer
+ logger.info("Loading SentenceTransformer model: %s", model_name)
+ _st_model = SentenceTransformer(model_name)
+ _st_model_name = model_name
+ return _st_model
+
+
+def compute_embedding(text: str, model: str = "all-MiniLM-L6-v2") -> List[float]:
+ st = _get_st_model(model)
+ vec = st.encode([text])[0]
+ return vec.tolist()
+
+
+def bm25_score(
+ query_tokens: List[str],
+ doc_tokens: List[str],
+ avg_doc_len: float,
+ doc_count: int,
+ df: Dict[str, int],
+ k1: float = 1.5,
+ b: float = 0.75,
+) -> float:
+ if not query_tokens or not doc_tokens:
+ return 0.0
+
+ doc_len = len(doc_tokens)
+ doc_tf = Counter(doc_tokens)
+ score = 0.0
+
+ for term in query_tokens:
+ if term not in doc_tf:
+ continue
+ tf = doc_tf[term]
+ n = df.get(term, 0)
+ idf = math.log((doc_count - n + 0.5) / (n + 0.5) + 1.0)
+ tf_norm = (tf * (k1 + 1)) / (tf + k1 * (1 - b + b * doc_len / max(avg_doc_len, 1)))
+ score += idf * tf_norm
+
+ return score
+
+
+def cosine_similarity(a: List[float], b: List[float]) -> float:
+ if not a or not b or len(a) != len(b):
+ return 0.0
+ dot = sum(x * y for x, y in zip(a, b))
+ norm_a = math.sqrt(sum(x * x for x in a))
+ norm_b = math.sqrt(sum(x * x for x in b))
+ if norm_a == 0 or norm_b == 0:
+ return 0.0
+ return dot / (norm_a * norm_b)
+
+
+def hybrid_search(
+ store: SkillStore,
+ query: str,
+ query_embedding: Optional[List[float]] = None,
+ top_k: int = 5,
+ threshold: float = 0.3,
+ bm25_weight: float = 0.1,
+ embedding_model: str = "text-embedding-3-small",
+) -> List[Tuple[SkillEntry, float]]:
+ skills = store.list_skills()
+ if not skills:
+ return []
+
+ query_tokens = tokenize(query)
+ doc_tokens_map: Dict[str, List[str]] = {}
+ df: Dict[str, int] = Counter()
+
+ for skill in skills:
+ tokens = tokenize(skill.to_search_text())
+ doc_tokens_map[skill.id] = tokens
+ for t in set(tokens):
+ df[t] += 1
+
+ avg_doc_len = sum(len(t) for t in doc_tokens_map.values()) / max(len(skills), 1)
+
+ bm25_scores: Dict[str, float] = {}
+ for skill in skills:
+ bm25_scores[skill.id] = bm25_score(
+ query_tokens, doc_tokens_map[skill.id],
+ avg_doc_len, len(skills), df,
+ )
+
+ vec_scores: Dict[str, float] = {}
+ if query_embedding:
+ embeddings = store.get_embeddings()
+ for skill in skills:
+ emb = embeddings.get(skill.id)
+ if emb:
+ vec_scores[skill.id] = cosine_similarity(query_embedding, emb)
+ else:
+ vec_scores[skill.id] = 0.0
+ else:
+ bm25_weight = 1.0
+ for skill in skills:
+ vec_scores[skill.id] = 0.0
+
+ bm25_max = max(bm25_scores.values()) if bm25_scores else 0.0
+ norm_bm25: Dict[str, float] = {}
+ if bm25_max > 0:
+ norm_bm25 = {k: v / bm25_max for k, v in bm25_scores.items()}
+ else:
+ norm_bm25 = {k: 0.0 for k in bm25_scores}
+
+ final_scores: Dict[str, float] = {}
+ for skill in skills:
+ sid = skill.id
+ final_scores[sid] = (
+ (1 - bm25_weight) * vec_scores.get(sid, 0.0)
+ + bm25_weight * norm_bm25.get(sid, 0.0)
+ )
+
+ skill_map = {s.id: s for s in skills}
+ results = [
+ (skill_map[sid], score)
+ for sid, score in sorted(final_scores.items(), key=lambda x: -x[1])
+ if score >= threshold
+ ]
+
+ return results[:top_k]
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/skill_store.py b/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/skill_store.py
new file mode 100644
index 00000000..a274925e
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/autoskill/skill_store.py
@@ -0,0 +1,208 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import json
+import threading
+import uuid
+from dataclasses import asdict, dataclass, field
+from datetime import datetime
+from typing import Any, Dict, List, Optional
+
+
+@dataclass
+class SkillEntry:
+
+ id: str
+ name: str
+ description: str
+ instructions: str
+ triggers: List[str] = field(default_factory=list)
+ tags: List[str] = field(default_factory=list)
+ examples: List[Dict[str, Any]] = field(default_factory=list)
+ version: str = "0.1.0"
+ confidence: float = 0.5
+ created_at: str = ""
+ updated_at: str = ""
+
+ def to_search_text(self) -> str:
+ parts = [self.name, self.description]
+ parts.extend(self.triggers)
+ parts.extend(self.tags)
+ parts.append(self.instructions)
+ return " ".join(parts)
+
+ def bump_version(self) -> None:
+ parts = self.version.split(".")
+ if len(parts) == 3:
+ parts[2] = str(int(parts[2]) + 1)
+ self.version = ".".join(parts)
+ else:
+ self.version = "0.1.1"
+ self.updated_at = datetime.now().isoformat()
+
+
+@dataclass
+class LearningEvent:
+
+ session_id: str
+ task_id: str
+ benchmark_id: str
+ action: str
+ skill_name: str
+ timestamp: str = ""
+
+
+class SkillStore:
+
+ _instances: Dict[str, "SkillStore"] = {}
+ _class_lock = threading.Lock()
+
+ def __init__(self, store_id: str) -> None:
+ self._store_id = store_id
+ self._lock = threading.Lock()
+ self._skills: Dict[str, SkillEntry] = {}
+ self._embeddings: Dict[str, List[float]] = {}
+ self._session_count: int = 0
+ self._history: List[LearningEvent] = []
+
+ @property
+ def store_id(self) -> str:
+ return self._store_id
+
+ @property
+ def session_count(self) -> int:
+ with self._lock:
+ return self._session_count
+
+ @property
+ def skill_count(self) -> int:
+ with self._lock:
+ return len(self._skills)
+
+ @classmethod
+ def get_or_create(
+ cls,
+ shuffle_mode: str = "isolated",
+ benchmark_id: Optional[str] = None,
+ task_group: Optional[str] = None,
+ ) -> "SkillStore":
+ if shuffle_mode == "isolated":
+ key = f"autoskill_isolated_{benchmark_id or task_group or 'default'}"
+ elif shuffle_mode == "sequential":
+ key = "autoskill_sequential_global"
+ elif shuffle_mode == "interleaved":
+ key = "autoskill_interleaved_global"
+ else:
+ key = f"autoskill_{shuffle_mode}"
+
+ with cls._class_lock:
+ if key not in cls._instances:
+ cls._instances[key] = cls(store_id=key)
+ return cls._instances[key]
+
+ @classmethod
+ def list_stores(cls) -> Dict[str, "SkillStore"]:
+ with cls._class_lock:
+ return dict(cls._instances)
+
+ @classmethod
+ def reset_all(cls) -> None:
+ with cls._class_lock:
+ cls._instances.clear()
+
+ def increment_session(self) -> int:
+ with self._lock:
+ self._session_count += 1
+ return self._session_count
+
+ def add_skill(self, skill: SkillEntry) -> None:
+ with self._lock:
+ if not skill.id:
+ skill.id = str(uuid.uuid4())
+ if not skill.created_at:
+ skill.created_at = datetime.now().isoformat()
+ skill.updated_at = skill.created_at
+ self._skills[skill.id] = skill
+
+ def update_skill(self, skill: SkillEntry) -> None:
+ with self._lock:
+ skill.updated_at = datetime.now().isoformat()
+ self._skills[skill.id] = skill
+
+ def list_skills(self) -> List[SkillEntry]:
+ with self._lock:
+ return list(self._skills.values())
+
+ def set_embedding(self, skill_id: str, embedding: List[float]) -> None:
+ with self._lock:
+ self._embeddings[skill_id] = embedding
+
+ def get_embeddings(self) -> Dict[str, List[float]]:
+ with self._lock:
+ return dict(self._embeddings)
+
+ def record_learning(
+ self,
+ session_id: str,
+ task_id: str,
+ benchmark_id: str,
+ action: str,
+ skill_name: str = "",
+ ) -> None:
+ with self._lock:
+ self._history.append(LearningEvent(
+ session_id=session_id,
+ task_id=task_id,
+ benchmark_id=benchmark_id,
+ action=action,
+ skill_name=skill_name,
+ timestamp=datetime.now().isoformat(),
+ ))
+
+ def save_checkpoint(self, path: str) -> None:
+ with self._lock:
+ data = {
+ "store_id": self._store_id,
+ "session_count": self._session_count,
+ "skills": {sid: asdict(s) for sid, s in self._skills.items()},
+ "embeddings": self._embeddings,
+ "history": [asdict(e) for e in self._history],
+ }
+ with open(path, "w", encoding="utf-8") as f:
+ json.dump(data, f, ensure_ascii=False, indent=2)
+
+ def load_checkpoint(self, path: str) -> None:
+ with open(path, "r", encoding="utf-8") as f:
+ data = json.load(f)
+ with self._lock:
+ self._session_count = data.get("session_count", 0)
+ self._skills = {}
+ for sid, sdata in data.get("skills", {}).items():
+ self._skills[sid] = SkillEntry(**{
+ k: v for k, v in sdata.items()
+ if k in SkillEntry.__dataclass_fields__
+ })
+ self._embeddings = data.get("embeddings", {})
+ self._history = [
+ LearningEvent(**{
+ k: v for k, v in e.items()
+ if k in LearningEvent.__dataclass_fields__
+ })
+ for e in data.get("history", [])
+ ]
+
+ def save_skills_text(self, path: str) -> None:
+ with self._lock:
+ skills = list(self._skills.values())
+ lines = [f"# SkillBank: {self._store_id} ({len(skills)} skills)\n"]
+ for s in skills:
+ lines.append(f"## {s.name} (v{s.version})")
+ lines.append(f" {s.description}")
+ lines.append(f" Tags: {', '.join(s.tags)}")
+ lines.append(f" Triggers: {', '.join(s.triggers)}")
+ lines.append(f" Instructions: {s.instructions}")
+ lines.append("")
+ with open(path, "w", encoding="utf-8") as f:
+ f.write("\n".join(lines))
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/cli/__init__.py b/labs/AgentStream/exgentic/src/exgentic/agents/cli/__init__.py
new file mode 100644
index 00000000..1aaf12d0
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/cli/__init__.py
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+# CLI agent bundle namespace
+
+from .claude.agent import ClaudeCodeAgent, ClaudeCodeAgentInstance # noqa: F401
+from .codex.agent import CodexAgent, CodexAgentInstance # noqa: F401
+from .command_runner import ExecutionBackend # noqa: F401
+from .gemini.agent import GeminiAgent, GeminiAgentInstance # noqa: F401
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/cli/base.py b/labs/AgentStream/exgentic/src/exgentic/agents/cli/base.py
new file mode 100644
index 00000000..395d0ab8
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/cli/base.py
@@ -0,0 +1,350 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from __future__ import annotations
+
+import abc
+import logging
+import os
+import tempfile
+from pathlib import Path
+from typing import Any
+
+from ...adapters.agents.mcp_agent import MCPAgentInstance
+from ...core.agent import Agent
+from ...core.context import context_env
+from ...core.types import ModelSettings
+from ...integrations.litellm import LitellmProxy
+from ...integrations.litellm.health import check_model_accessible_sync
+from ...integrations.litellm.trace_cost import load_trace_cost
+from ...utils.cost import UpdatableCostReport
+from .command_runner import (
+ BaseCLIConfig,
+ CLIResult,
+ DockerRunner,
+ ExecutionBackend,
+ PodmanRunner,
+ ProcessRunner,
+)
+
+
+class BaseCLIWrapper(abc.ABC):
+ """Shared helpers for headless CLI wrappers.
+
+ Subclasses implement build_env/build_command; run() handles the rest.
+ """
+
+ config_prefix = "cli_"
+ spawn_error_message = "Spawn failed"
+
+ def __init__(
+ self,
+ env: dict[str, str] | None = None,
+ log_path: Path | None = None,
+ config_dir: Path | None = None,
+ logger: logging.Logger | None = None,
+ runner: ExecutionBackend = ExecutionBackend.AUTO,
+ ) -> None:
+ self.env = env or os.environ.copy()
+ self.config_dir = config_dir
+ self.log_path = log_path
+ self._last_run_context: dict[str, Any] = {}
+ self._logger = logger or logging.getLogger(self.__class__.__name__)
+ if runner == ExecutionBackend.AUTO:
+ from .command_runner import resolve_container_backend
+
+ runner = resolve_container_backend()
+ if runner == ExecutionBackend.PROCESS:
+ self.runner = ProcessRunner(log_path=log_path, logger=self._logger)
+ elif runner == ExecutionBackend.PODMAN:
+ self.runner = PodmanRunner(log_path=log_path, logger=self._logger)
+ elif runner == ExecutionBackend.DOCKER:
+ self.runner = DockerRunner(log_path=log_path, logger=self._logger)
+ else:
+ raise ValueError(f"runner value: {runner} is not supported!")
+
+ def run(self, *, prompt: str, config: BaseCLIConfig) -> CLIResult:
+ cfg_root = self._resolve_config_root(self.config_prefix)
+ self._last_run_context = {
+ "cfg_root": cfg_root,
+ "prompt": prompt,
+ "config": config,
+ }
+ env = self.build_env(cfg_root=cfg_root, prompt=prompt, config=config)
+ env.update(context_env())
+ if config.env:
+ env = {**env, **config.env}
+ cmd = self.build_command(cfg_root=cfg_root, prompt=prompt, config=config)
+ return self.runner.run(
+ cmd=cmd,
+ env=env,
+ cfg_root=cfg_root,
+ config=config,
+ spawn_error_message=self.spawn_error_message,
+ )
+
+ def close(self) -> None:
+ self.runner.close()
+
+ def _resolve_config_root(self, prefix: str) -> Path:
+ if self.config_dir is not None:
+ return Path(self.config_dir)
+ return Path(tempfile.mkdtemp(prefix=prefix))
+
+ def _log_warning(self, message: str) -> None:
+ """Best-effort logger wrapper to avoid attribute errors on exit."""
+ try:
+ self._logger.warning(message)
+ except Exception:
+ logging.getLogger(__name__).warning(message)
+
+ # Abstract hooks -------------------------------------------------
+
+ @abc.abstractmethod
+ def build_env(self, *, cfg_root: Path, prompt: str, config: Any) -> dict[str, str]:
+ ...
+
+ @abc.abstractmethod
+ def build_command(self, *, cfg_root: Path, prompt: str, config: Any) -> list[str]:
+ ...
+
+
+class ProxyBackedMCPAgentInstance(MCPAgentInstance, abc.ABC):
+ """Base class for MCP agents that launch a LiteLLM proxy and delegate to a CLI wrapper."""
+
+ def __init__(
+ self,
+ session_id: str,
+ model_id: str,
+ *,
+ max_steps: int = 150,
+ model_alias: str | None = None,
+ execution_backend: ExecutionBackend = ExecutionBackend.AUTO,
+ model_settings: ModelSettings | None = None,
+ ) -> None:
+ super().__init__(session_id)
+ self.model_id = model_id
+ self.max_steps = max_steps
+ self._proxy_log_dir = self.paths.agent_dir / "litellm_proxy"
+ self._trace_log_path = self._proxy_log_dir / "trace.jsonl"
+ self._proxy: LitellmProxy | None = None
+ self._cli: BaseCLIWrapper | None = None
+ self._model_alias = model_alias
+ self.execution_backend = execution_backend
+ if model_settings is None:
+ self.model_settings = ModelSettings()
+ elif isinstance(model_settings, ModelSettings):
+ self.model_settings = model_settings
+ else:
+ raise ValueError("model_settings must be a ModelSettings instance.")
+
+ # Check model accessibility
+ check_model_accessible_sync(self.model_id, logger=self.logger)
+
+ @property
+ @abc.abstractmethod
+ def cli_display_name(self) -> str:
+ ...
+
+ @abc.abstractmethod
+ def _build_cli(self) -> BaseCLIWrapper:
+ ...
+
+ @abc.abstractmethod
+ def _run_cli(
+ self,
+ cli: BaseCLIWrapper,
+ prompt: str,
+ mcp_host: str,
+ mcp_port: int,
+ proxy: LitellmProxy,
+ ) -> Any:
+ ...
+
+ def close_mcp_agent(self) -> None:
+ self.logger.info("Closing CLI before MCP shutdown")
+ if self._cli is not None:
+ self._cli.close()
+ self._cli = None
+ self.logger.info("Closing LiteLLM proxy before MCP shutdown")
+ if self._proxy is not None:
+ self._proxy.close()
+ self._proxy = None
+ super().close_mcp_agent()
+
+ def run_mcp_agent(self, mcp_host: str, mcp_port: int) -> Any:
+ prompt = self._build_prompt()
+ proxy_log = self._proxy_log_dir / "litellm_proxy.log"
+ alias_map = self._proxy_alias_map()
+
+ # Log model parameters for validation
+ params_info = []
+ if self.model_settings.temperature is not None:
+ params_info.append(f"temperature={self.model_settings.temperature}")
+ if self.model_settings.max_tokens is not None:
+ params_info.append(f"max_tokens={self.model_settings.max_tokens}")
+ if self.model_settings.top_p is not None:
+ params_info.append(f"top_p={self.model_settings.top_p}")
+ params_str = f" with {', '.join(params_info)}" if params_info else ""
+
+ self.logger.info(
+ "Starting LiteLLM proxy for model %s%s (log: %s)",
+ self.model_id,
+ params_str,
+ proxy_log,
+ )
+ proxy_log.parent.mkdir(parents=True, exist_ok=True)
+ proxy = LitellmProxy(
+ model=self.model_id,
+ log_path=str(proxy_log),
+ usage_log_path=str(self._trace_log_path),
+ model_alias_map=alias_map or None,
+ model_settings=self.model_settings,
+ )
+ self._proxy = proxy
+ try:
+ proxy.start()
+ except Exception:
+ self.logger.exception("LiteLLM proxy failed to start")
+ raise
+ self.logger.info("LiteLLM proxy started at %s", proxy.base_url)
+
+ try:
+ cli = self._build_cli()
+ self._cli = cli
+ self.logger.info(
+ "Launching %s (log: %s) against MCP http://%s:%s/mcp using proxy %s",
+ self.cli_display_name,
+ cli.log_path,
+ mcp_host,
+ mcp_port,
+ proxy.base_url,
+ )
+ stdout = self._run_cli(cli, prompt, mcp_host, mcp_port, proxy)
+ self.logger.info("%s run finished", self.cli_display_name)
+ return stdout
+ except Exception as e:
+ from .command_runner import CLIExecutionError
+
+ if isinstance(e, CLIExecutionError):
+ if e.stderr:
+ self.logger.error("%s STDERR:\n%s", self.cli_display_name, e.stderr.rstrip())
+ if e.stdout:
+ self.logger.error("%s STDOUT:\n%s", self.cli_display_name, e.stdout.rstrip())
+ self.logger.exception("%s run failed: %s", self.cli_display_name, e)
+ raise
+ finally:
+ if self._cli is not None:
+ self._cli.close()
+ self._cli = None
+ proxy.close()
+ self._proxy = None
+ self._drain_server()
+
+ def get_cost(self) -> UpdatableCostReport:
+ cost = load_trace_cost(self._trace_log_path, self.model_id)
+ report = UpdatableCostReport.initialize_empty(model_name=self.model_id)
+ report.add_cost(cost)
+ return report
+
+ def _build_prompt(self) -> str:
+ prompt = ""
+ if self.context:
+ prompt += f"Context: {self.context}\n\n"
+
+ finish_hint = ""
+ finish_tools = [a.name for a in self.actions if a.is_finish]
+ if finish_tools:
+ finish_hint = f" Use the designated finish tool(s): {', '.join(finish_tools)}."
+
+ instructions = (
+ "Complete this task using the available environment tools. Each tool corresponds to an action "
+ " you can take in the task environment.\n"
+ "# Important: You are on solo mode. Do not reply back or message unless its through a dedicated "
+ "environment tool call, every such attempt will finish the session with failure.\n"
+ "All your actions on with regard to the task must go through environment tool calls."
+ )
+
+ if finish_hint:
+ instructions += (
+ f"{finish_hint} Always conclude by invoking the designated finish tool for this task environment."
+ )
+ prompt += f"{instructions}\n"
+
+ if self.initial_observation is not None and not self.initial_observation.is_empty():
+ text = str(self.initial_observation).strip()
+ if text:
+ prompt += f"\nFirst Observation: {text}\n"
+
+ return prompt + self.task
+
+ def close(self) -> None:
+ if self._cli is not None:
+ self._cli.close()
+ self._cli = None
+ if self._proxy is not None:
+ self._proxy.close()
+ self._proxy = None
+ self._drain_server()
+ super().close()
+
+ def _proxy_alias_map(self) -> dict[str, str]:
+ if not self._model_alias:
+ return {}
+ return {self._model_alias: self.model_id}
+
+ def _drain_server(self) -> None:
+ """Best-effort stop/join of the MCP server thread to avoid teardown crashes."""
+ server = self._mcp_server
+ if server is None:
+ return
+ try:
+ server.stop(raise_on_timeout=False)
+ except Exception as exc:
+ self._log_warning(f"Error while stopping MCP server: {exc}")
+
+ def _log_warning(self, message: str) -> None:
+ try:
+ self._logger.warning(message)
+ except Exception:
+ logging.getLogger(__name__).warning(message)
+
+
+class ProxyBackedAgent(Agent):
+ """Minimal agent factory helper for proxy-backed CLI agents."""
+
+ model: str
+ max_steps: int = 150
+
+ @classmethod
+ def _get_instance_class(cls):
+ raise NotImplementedError
+
+ execution_backend: ExecutionBackend = ExecutionBackend.AUTO
+ model_settings: ModelSettings | None = None
+
+ def _get_instance_kwargs(
+ self,
+ session_id: str,
+ ) -> dict[str, Any]:
+ # Resolve AUTO on the host side so the concrete backend (PODMAN/DOCKER)
+ # is serialized to the venv, where podman may not be on PATH.
+ backend = self.execution_backend
+ if backend == ExecutionBackend.AUTO:
+ from .command_runner import resolve_container_backend
+
+ backend = resolve_container_backend()
+ return {
+ "session_id": session_id,
+ "model_id": self.model,
+ "max_steps": self.max_steps,
+ "execution_backend": backend,
+ "model_settings": self.model_settings,
+ }
+
+ @property
+ def model_name(self) -> str: # type: ignore[override]
+ return str(self.model).split("/")[-1]
+
+ def get_models_names(self) -> list[str]: # type: ignore[override]
+ return [str(self.model)]
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/cli/claude/__init__.py b/labs/AgentStream/exgentic/src/exgentic/agents/cli/claude/__init__.py
new file mode 100644
index 00000000..95e9b24a
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/cli/claude/__init__.py
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from .agent import ClaudeCodeAgent, ClaudeCodeAgentInstance # noqa: F401
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/cli/claude/agent.py b/labs/AgentStream/exgentic/src/exgentic/agents/cli/claude/agent.py
new file mode 100644
index 00000000..220fb534
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/cli/claude/agent.py
@@ -0,0 +1,93 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from __future__ import annotations
+
+import os
+from typing import Any, ClassVar
+
+from ....core.types import ModelSettings
+from ..base import ExecutionBackend, ProxyBackedAgent, ProxyBackedMCPAgentInstance
+from .cli import ClaudeCLIConfig, ClaudeCodeCLI
+
+
+class ClaudeCodeAgentInstance(ProxyBackedMCPAgentInstance):
+ """Self-contained Claude Code CLI agent routed through a LiteLLM proxy."""
+
+ def __init__(
+ self,
+ session_id: str,
+ model_id: str,
+ max_steps: int = 150,
+ execution_backend: ExecutionBackend = ExecutionBackend.AUTO,
+ model_settings: ModelSettings | None = None,
+ ):
+ # The alias is what we ask Claude Code CLI for; the proxy maps it to the backend model.
+ # Must contain "sonnet-4" to get 64000 max_output_tokens in CLI
+ # (CLI hardcodes 8192 for names containing "3-5").
+ self._claude_model_alias = "claude-sonnet-4-20250514"
+ super().__init__(
+ session_id,
+ model_id,
+ max_steps=max_steps,
+ model_alias=self._claude_model_alias,
+ execution_backend=execution_backend,
+ model_settings=model_settings,
+ )
+ self._claude_log = self.paths.agent_dir / "claude_cli.log"
+
+ @property
+ def cli_display_name(self) -> str:
+ return "Claude Code CLI"
+
+ def _build_cli(self) -> ClaudeCodeCLI:
+ cfg_dir = self.paths.agent_dir / "claude_code_config"
+ return ClaudeCodeCLI(
+ env=os.environ.copy(),
+ log_path=self._claude_log,
+ config_dir=cfg_dir,
+ logger=self.logger,
+ runner=self.execution_backend,
+ )
+
+ def _run_cli(
+ self,
+ cli: ClaudeCodeCLI,
+ prompt: str,
+ mcp_host: str,
+ mcp_port: int,
+ proxy: Any,
+ ) -> Any:
+ # allowed_tools = [f"mcp__environment__{action.name}" for action in self.actions]
+ config = ClaudeCLIConfig(
+ mcp_host=mcp_host,
+ mcp_port=mcp_port,
+ provider_url=proxy.base_url,
+ backend_model=self.model_id,
+ claude_model=self._claude_model_alias,
+ # allowed_tools=allowed_tools,
+ max_turns=self.max_steps,
+ )
+ config.env = {
+ "MCP_TIMEOUT": str(config.mcp_timeout_ms),
+ "MCP_TOOL_TIMEOUT": str(config.mcp_tool_timeout_ms),
+ "CLAUDE_CODE_MAX_OUTPUT_TOKENS": "32768",
+ }
+ result = cli.run(prompt=prompt, config=config)
+ return result.stdout
+
+ def _stringify_empty_output(self) -> bool:
+ return True
+
+
+class ClaudeCodeAgent(ProxyBackedAgent):
+ display_name: ClassVar[str] = "Claude Code CLI"
+ slug_name: ClassVar[str] = "claude_code"
+ execution_backend: ExecutionBackend = ExecutionBackend.AUTO
+
+ @classmethod
+ def _get_instance_class(cls):
+ return ClaudeCodeAgentInstance
+
+ def get_models_names(self) -> list[str]: # type: ignore[override]
+ return [str(self.model_id)]
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/cli/claude/cli.py b/labs/AgentStream/exgentic/src/exgentic/agents/cli/claude/cli.py
new file mode 100644
index 00000000..225532f9
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/cli/claude/cli.py
@@ -0,0 +1,148 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Optional
+
+from ..base import BaseCLIConfig, BaseCLIWrapper, ExecutionBackend
+
+
+class ClaudeCLIConfig(BaseCLIConfig):
+ backend_model: str
+ claude_model: str
+ env_key: str = "ANTHROPIC_API_KEY"
+ auth_token_env: str = "ANTHROPIC_AUTH_TOKEN"
+ output_format: str = "text"
+ skip_permissions: bool = True
+ max_turns: int = 150
+ mcp_only: bool = True # New: only allow MCP tools
+ allowed_tools: Optional[list[str]] = None
+ image: str = "exgentic-claude-code:dev"
+ image_workdir: str = "/work"
+ mcp_timeout_ms: int = 600_000
+ mcp_tool_timeout_ms: int = 1_800_000
+
+
+class ClaudeCodeCLI(BaseCLIWrapper):
+ """Thin wrapper for running Claude Code's CLI in print mode."""
+
+ def __init__(
+ self,
+ env: Optional[dict[str, str]] = None,
+ log_path: Optional[Path] = None,
+ config_dir: Optional[Path] = None,
+ logger=None,
+ runner: ExecutionBackend = ExecutionBackend.PROCESS,
+ ) -> None:
+ super().__init__(
+ env=env,
+ log_path=log_path,
+ config_dir=config_dir,
+ logger=logger,
+ runner=runner,
+ )
+ self._mcp_config_path: Optional[Path] = None
+ self.config_prefix = "claude_code_cli_"
+ self.spawn_error_message = "Failed to start Claude CLI"
+
+ # Required hooks --------------------------------------------------
+
+ def build_env(self, *, cfg_root: Path, prompt: str, config: ClaudeCLIConfig) -> dict[str, str]:
+ env = self.env.copy()
+ env["ANTHROPIC_BASE_URL"] = config.provider_url
+ if config.env:
+ env.update(config.env)
+
+ token = env.get(config.auth_token_env) or env.get(config.env_key) or "dummy-api-key"
+ env[config.env_key] = token
+ env[config.auth_token_env] = token
+
+ # Isolate config/settings/state under a temp directory
+ env["HOME"] = str(cfg_root)
+
+ return env
+
+ def build_command(self, *, cfg_root: Path, prompt: str, config: ClaudeCLIConfig) -> list[str]:
+ if not prompt or not prompt.strip():
+ raise ValueError("Prompt cannot be empty")
+
+ mcp_cfg_path = (cfg_root / "mcp.json").absolute()
+ self._mcp_config_path = mcp_cfg_path
+
+ # Rewrite localhost addresses to host gateway for container runners
+ mcp_host = config.mcp_host
+ from ..command_runner import ContainerRunner
+
+ if isinstance(self.runner, ContainerRunner) and mcp_host in ("0.0.0.0", "127.0.0.1", "localhost"):
+ mcp_host = self.runner.host_gateway
+
+ mcp_url = f"http://{mcp_host}:{config.mcp_port}/mcp"
+ self._write_mcp_config(mcp_cfg_path, mcp_url)
+ self._write_settings_config(cfg_root)
+
+ # Verify config was written
+ if not mcp_cfg_path.exists():
+ raise RuntimeError(f"Failed to write MCP config to {mcp_cfg_path}")
+
+ cmd: list[str] = [
+ "claude",
+ "-p", # print mode, single-shot
+ "--model",
+ config.claude_model,
+ "--mcp-config",
+ str(mcp_cfg_path),
+ "--strict-mcp-config",
+ "--output-format",
+ config.output_format,
+ "--debug",
+ "--mcp-debug",
+ "--no-session-persistence",
+ "--append-system-prompt",
+ "AUTONOMOUS SOLO MODE.\n"
+ "First, discover available MCP tools by listing tools from the environment server.\n"
+ "Then use those tools to complete the task.\n"
+ "Never ask for clarification - make reasonable assumptions.\n"
+ "You have NO filesystem access - work exclusively through MCP tools.",
+ "--dangerously-skip-permissions",
+ "--max-turns",
+ str(config.max_turns),
+ ]
+
+ # if config.allowed_tools:
+ # cmd.extend(["--allowedTools", ",".join(config.allowed_tools)])
+
+ # Ensure the following argument is treated purely as the prompt, not part of a variadic flag.
+ cmd.append("--")
+ cmd.append(prompt)
+ return cmd
+
+ # Internal helpers -------------------------------------------------
+
+ def _write_mcp_config(self, path: Path, mcp_url: str) -> None:
+ """Write MCP server configuration."""
+ config = {"mcpServers": {"environment": {"type": "http", "url": mcp_url}}}
+ path.parent.mkdir(parents=True, exist_ok=True)
+ with open(path, "w", encoding="utf-8") as fh:
+ json.dump(config, fh, indent=2)
+
+ def _write_settings_config(self, home_dir: Path) -> None:
+ """Write settings that completely block filesystem access."""
+ settings_path = home_dir / ".claude" / "settings.json"
+
+ settings = {
+ "enableAllProjectMcpServers": True,
+ }
+
+ settings_path.parent.mkdir(parents=True, exist_ok=True)
+ with open(settings_path, "w", encoding="utf-8") as fh:
+ json.dump(settings, fh, indent=2)
+
+ # Pre-create directories that the Claude Code CLI expects to write
+ # into. When running inside a container with ``--user``, the
+ # mounted volume may have restrictive ownership so the CLI cannot
+ # create these itself.
+ for subdir in ("debug", "conversations", "projects", "todos"):
+ (home_dir / ".claude" / subdir).mkdir(parents=True, exist_ok=True)
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/cli/claude/setup.sh b/labs/AgentStream/exgentic/src/exgentic/agents/cli/claude/setup.sh
new file mode 100644
index 00000000..dc85320f
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/cli/claude/setup.sh
@@ -0,0 +1,32 @@
+#!/bin/bash
+set -euo pipefail
+
+# Determine container runtime
+CONTAINER_CMD=""
+if command -v podman >/dev/null 2>&1; then
+ CONTAINER_CMD="podman"
+ # Start podman machine if needed (macOS/Windows)
+ if podman machine list >/dev/null 2>&1; then
+ MACHINE_STATUS=$(podman machine list --format "{{.Running}}" 2>/dev/null | head -n 1)
+ if [ -z "$MACHINE_STATUS" ]; then
+ podman machine init && podman machine start
+ elif [ "$MACHINE_STATUS" != "true" ]; then
+ podman machine start
+ fi
+ fi
+elif command -v docker >/dev/null 2>&1; then
+ CONTAINER_CMD="docker"
+else
+ echo "Error: Neither Podman nor Docker found." >&2
+ exit 1
+fi
+
+# Build Claude Code container image (inline — no external Dockerfile needed)
+$CONTAINER_CMD build -t exgentic-claude-code:dev -f - . <<'DOCKERFILE'
+FROM registry.access.redhat.com/ubi9/nodejs-20
+RUN npm install -g @anthropic-ai/claude-code@2.1.7
+WORKDIR /work
+CMD ["claude","--help"]
+DOCKERFILE
+
+echo "Claude Code Agent setup complete (using $CONTAINER_CMD)"
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/cli/codex/__init__.py b/labs/AgentStream/exgentic/src/exgentic/agents/cli/codex/__init__.py
new file mode 100644
index 00000000..38ee1b99
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/cli/codex/__init__.py
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from .agent import CodexAgent, CodexAgentInstance # noqa: F401
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/cli/codex/agent.py b/labs/AgentStream/exgentic/src/exgentic/agents/cli/codex/agent.py
new file mode 100644
index 00000000..0f9e5a98
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/cli/codex/agent.py
@@ -0,0 +1,71 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from __future__ import annotations
+
+import os
+from typing import Any, ClassVar
+
+from ....core.types import ModelSettings
+from ..base import ProxyBackedAgent, ProxyBackedMCPAgentInstance
+from .cli import CodexCLI, CodexCLIConfig, ExecutionBackend
+
+
+class CodexAgentInstance(ProxyBackedMCPAgentInstance):
+ """Self-contained Codex CLI agent that runs through a LiteLLM proxy."""
+
+ def __init__(
+ self,
+ session_id: str,
+ model_id: str,
+ max_steps: int = 150,
+ execution_backend: ExecutionBackend = ExecutionBackend.AUTO,
+ model_settings: ModelSettings | None = None,
+ ):
+ super().__init__(
+ session_id,
+ model_id,
+ max_steps=max_steps,
+ execution_backend=execution_backend,
+ model_settings=model_settings,
+ )
+ self._codex_log = self.paths.agent_dir / "codex_cli.log"
+
+ @property
+ def cli_display_name(self) -> str:
+ return "Codex CLI"
+
+ def _build_cli(self) -> CodexCLI:
+ return CodexCLI(
+ env=os.environ.copy(),
+ log_path=self._codex_log,
+ logger=self.logger,
+ runner=self.execution_backend,
+ )
+
+ def _run_cli(
+ self,
+ cli: CodexCLI,
+ prompt: str,
+ mcp_host: str,
+ mcp_port: int,
+ proxy: Any,
+ ) -> Any:
+ config = CodexCLIConfig(
+ mcp_host=mcp_host,
+ mcp_port=mcp_port,
+ model_id=self.model_id,
+ provider_url=proxy.base_url,
+ )
+ result = cli.run(prompt=prompt, config=config)
+ return result.stdout
+
+
+class CodexAgent(ProxyBackedAgent):
+ display_name: ClassVar[str] = "Codex CLI"
+ slug_name: ClassVar[str] = "codex_cli"
+ execution_backend: ExecutionBackend = ExecutionBackend.AUTO
+
+ @classmethod
+ def _get_instance_class(cls):
+ return CodexAgentInstance
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/cli/codex/cli.py b/labs/AgentStream/exgentic/src/exgentic/agents/cli/codex/cli.py
new file mode 100644
index 00000000..1d15a2e7
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/cli/codex/cli.py
@@ -0,0 +1,70 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Optional
+
+from ..base import BaseCLIConfig, BaseCLIWrapper, ExecutionBackend
+
+
+class CodexCLIConfig(BaseCLIConfig):
+ model_id: str
+ env_key: str = "OPENAI_API_KEY"
+ profile: str = "temp_model"
+ provider_name: str = "Provider"
+ image: str = "exgentic-codex:dev"
+
+
+class CodexCLI(BaseCLIWrapper):
+ """Standalone wrapper for launching Codex CLI."""
+
+ def __init__(
+ self,
+ env: Optional[dict[str, str]] = None,
+ log_path: Optional[Path] = None,
+ logger=None,
+ runner: ExecutionBackend = ExecutionBackend.PROCESS,
+ ) -> None:
+ super().__init__(env=env, log_path=log_path, config_dir=None, logger=logger, runner=runner)
+ self.config_prefix = "codex_cli_"
+ self.spawn_error_message = "Failed to start Codex CLI"
+
+ def build_env(self, *, cfg_root: Path, prompt: str, config: CodexCLIConfig) -> dict[str, str]:
+ env = self.env.copy()
+ env["OPENAI_API_BASE"] = config.provider_url
+ api_key = env.get(config.env_key) or "dummy-api-key"
+ env[config.env_key] = api_key
+ return env
+
+ def build_command(self, *, cfg_root: Path, prompt: str, config: CodexCLIConfig) -> list[str]:
+ # Rewrite localhost addresses to host gateway for container runners
+ mcp_host = config.mcp_host
+ provider_url = config.provider_url
+ from ..command_runner import ContainerRunner
+
+ if isinstance(self.runner, ContainerRunner):
+ if mcp_host in ("0.0.0.0", "127.0.0.1", "localhost"):
+ mcp_host = self.runner.host_gateway
+ for local in ("://127.0.0.1:", "://localhost:"):
+ if local in provider_url:
+ provider_url = provider_url.replace(local, f"://{self.runner.host_gateway}:")
+ break
+
+ mcp_url = f"http://{mcp_host}:{config.mcp_port}/mcp"
+ overrides = [
+ f'mcp_servers.environment.url="{mcp_url}"',
+ f'model_providers.temp.name="{config.provider_name}"',
+ f'model_providers.temp.base_url="{provider_url}"',
+ f'model_providers.temp.env_key="{config.env_key}"',
+ f'profiles.{config.profile}.model_provider="temp"',
+ f'profiles.{config.profile}.model="{config.model_id}"',
+ ]
+
+ cmd: list[str] = ["codex", "exec", "--skip-git-repo-check", "--profile", config.profile]
+ for override in overrides:
+ cmd.extend(["-c", override])
+ cmd.append(prompt)
+
+ return cmd
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/cli/codex/setup.sh b/labs/AgentStream/exgentic/src/exgentic/agents/cli/codex/setup.sh
new file mode 100644
index 00000000..80c8e0fd
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/cli/codex/setup.sh
@@ -0,0 +1,32 @@
+#!/bin/bash
+set -euo pipefail
+
+# Determine container runtime
+CONTAINER_CMD=""
+if command -v podman >/dev/null 2>&1; then
+ CONTAINER_CMD="podman"
+ # Start podman machine if needed (macOS/Windows)
+ if podman machine list >/dev/null 2>&1; then
+ MACHINE_STATUS=$(podman machine list --format "{{.Running}}" 2>/dev/null | head -n 1)
+ if [ -z "$MACHINE_STATUS" ]; then
+ podman machine init && podman machine start
+ elif [ "$MACHINE_STATUS" != "true" ]; then
+ podman machine start
+ fi
+ fi
+elif command -v docker >/dev/null 2>&1; then
+ CONTAINER_CMD="docker"
+else
+ echo "Error: Neither Podman nor Docker found." >&2
+ exit 1
+fi
+
+# Build Codex CLI container image (inline — no external Dockerfile needed)
+$CONTAINER_CMD build -t exgentic-codex:dev -f - . <<'DOCKERFILE'
+FROM registry.access.redhat.com/ubi9/nodejs-20
+RUN npm install -g @openai/codex@0.93.0
+WORKDIR /work
+CMD ["codex","--help"]
+DOCKERFILE
+
+echo "Codex Agent setup complete (using $CONTAINER_CMD)"
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/cli/command_runner.py b/labs/AgentStream/exgentic/src/exgentic/agents/cli/command_runner.py
new file mode 100644
index 00000000..3dd2a26d
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/cli/command_runner.py
@@ -0,0 +1,471 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from __future__ import annotations
+
+import json
+import os
+import shlex
+import shutil
+import subprocess
+from dataclasses import dataclass
+from enum import Enum
+from pathlib import Path
+from typing import Optional
+
+from pydantic import BaseModel
+
+
+class ExecutionBackend(str, Enum):
+ """Execution backend for CLI runs."""
+
+ PROCESS = "process"
+ PODMAN = "podman"
+ DOCKER = "docker"
+ AUTO = "auto"
+
+
+def resolve_container_backend() -> ExecutionBackend:
+ """Auto-detect container runtime: prefer podman, fallback to docker."""
+ if shutil.which("podman"):
+ return ExecutionBackend.PODMAN
+ if shutil.which("docker"):
+ return ExecutionBackend.DOCKER
+ raise RuntimeError("Neither podman nor docker found")
+
+
+@dataclass
+class CLIResult:
+ stdout: str
+ stderr: str
+ code: int
+
+
+class CLIStartError(RuntimeError):
+ """Raised when a CLI fails to spawn."""
+
+
+class CLIExecutionError(RuntimeError):
+ """Raised when a CLI exits with a non-zero status."""
+
+ def __init__(
+ self,
+ message: str,
+ *,
+ code: int,
+ stdout: str,
+ stderr: str,
+ cmd: list[str],
+ ) -> None:
+ super().__init__(message)
+ self.code = code
+ self.stdout = stdout
+ self.stderr = stderr
+ self.cmd = cmd
+
+ def __str__(self) -> str:
+ parts = [super().__str__()]
+ if self.stderr:
+ parts.append(f"STDERR:\n{self.stderr.rstrip()}")
+ if self.stdout:
+ parts.append(f"STDOUT:\n{self.stdout.rstrip()}")
+ return "\n".join(parts)
+
+
+class BaseCLIConfig(BaseModel):
+ """Common config fields shared by CLI wrappers."""
+
+ mcp_host: str
+ mcp_port: int
+ provider_url: str
+ image: str
+ image_workdir: str = "/work"
+ env: Optional[dict[str, str]] = None
+
+
+class ProcessRunner:
+ """Shared subprocess execution logic (spawn + communicate + timeout + kill).
+
+ Concrete runners implement how cmd/env/cfg_root are transformed.
+ """
+
+ def __init__(self, log_path, logger):
+ super().__init__()
+ self.log_path = log_path
+ self._logger = logger
+ self._last_cmd: list[str] = []
+ self._proc: Optional[subprocess.Popen[str]] = None
+
+ def _write_log(
+ self,
+ stdout: str,
+ stderr: str,
+ *,
+ returncode: int,
+ config: BaseCLIConfig,
+ ) -> None:
+ if not self.log_path:
+ return
+ self.log_path.parent.mkdir(parents=True, exist_ok=True)
+ try:
+ config_json = config.model_dump_json(indent=2)
+ except Exception:
+ config_json = str(config)
+ with open(self.log_path, "w", encoding="utf-8") as fh:
+ fh.write(f"Command: {shlex.join(self._last_cmd)}\n")
+ fh.write("Config:\n")
+ fh.write(f"{config_json}\n")
+ fh.write(f"Exit code: {returncode}\n\n")
+ if stdout:
+ fh.write("STDOUT:\n")
+ fh.write(stdout)
+ if not stdout.endswith("\n"):
+ fh.write("\n")
+ if stderr:
+ if stdout:
+ fh.write("\n")
+ fh.write("STDERR:\n")
+ fh.write(stderr)
+ if not stderr.endswith("\n"):
+ fh.write("\n")
+
+ def run(
+ self,
+ *,
+ cmd: list[str],
+ env: dict[str, str],
+ cfg_root: Path,
+ config: BaseCLIConfig,
+ spawn_error_message: str,
+ stdin_devnull: bool = False,
+ ) -> CLIResult:
+ self._last_cmd = cmd
+ stdout: str = ""
+ stderr: str = ""
+ code: int = -1
+
+ popen_stdin = subprocess.DEVNULL if stdin_devnull else None
+ timeout_s: Optional[float] = None
+
+ try:
+ self._proc = subprocess.Popen(
+ cmd,
+ stdin=popen_stdin,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.PIPE,
+ text=True,
+ env=env,
+ )
+ except Exception as exc:
+ stderr = f"{spawn_error_message}: {exc}"
+ self._write_log(stdout, stderr, returncode=code, config=config)
+ raise CLIStartError(f"{spawn_error_message}: {exc}") from exc
+
+ try:
+ try:
+ stdout, stderr = self._proc.communicate(timeout=timeout_s)
+ except subprocess.TimeoutExpired:
+ self._logger.warning("CLI timed out; terminating: %s", shlex.join(cmd))
+ self._proc.terminate()
+ try:
+ stdout, stderr = self._proc.communicate(timeout=5)
+ except subprocess.TimeoutExpired:
+ self._logger.warning("CLI did not terminate; killing it: %s", shlex.join(cmd))
+ self._proc.kill()
+ stdout, stderr = self._proc.communicate()
+ code = self._proc.returncode or 0
+ finally:
+ self._write_log(stdout or "", stderr or "", returncode=code, config=config)
+ if code != 0:
+ self._logger.warning("CLI exited non-zero (%s): %s", code, shlex.join(cmd))
+ if code != 0:
+ raise CLIExecutionError(
+ f"CLI exited non-zero ({code}): {shlex.join(cmd)}",
+ code=code,
+ stdout=stdout or "",
+ stderr=stderr or "",
+ cmd=cmd,
+ )
+ return CLIResult(stdout=stdout or "", stderr=stderr or "", code=code)
+
+ def close(self) -> None:
+ proc = self._proc
+ if self._proc and self._proc.poll() is None:
+ self._proc.terminate()
+ try:
+ self._proc.wait(timeout=5)
+ except subprocess.TimeoutExpired:
+ self._logger.warning("CLI process did not terminate; killing it.")
+ self._proc.kill()
+ try:
+ proc.wait(timeout=5)
+ except subprocess.TimeoutExpired:
+ self._logger.warning("CLI process did not exit after kill.")
+
+
+class ContainerRunner(ProcessRunner):
+ """Shared helpers for container-based runners."""
+
+ host_gateway: str = "host.docker.internal"
+
+ def _rewrite_mcp_config_path(self, inner_cmd: list[str], workdir: str) -> list[str]:
+ if "--mcp-config" in inner_cmd:
+ i = inner_cmd.index("--mcp-config")
+ if i + 1 < len(inner_cmd):
+ inner_cmd[i + 1] = f"{workdir}/mcp.json"
+ return inner_cmd
+
+ def _container_env_from(
+ self,
+ env: dict[str, str],
+ host_gateway: str,
+ config: BaseCLIConfig,
+ ) -> dict[str, str]:
+ forwarded: dict[str, str] = {}
+
+ for k in (
+ # Anthropic (Claude Code)
+ "ANTHROPIC_BASE_URL",
+ "ANTHROPIC_API_KEY",
+ "ANTHROPIC_AUTH_TOKEN",
+ # OpenAI (Codex)
+ "OPENAI_API_KEY",
+ "OPENAI_API_BASE",
+ # Google (Gemini)
+ "GEMINI_API_KEY",
+ "GOOGLE_GEMINI_BASE_URL",
+ ):
+ if k in env:
+ forwarded[k] = env[k]
+ for k, v in env.items():
+ if k.startswith("EXGENTIC_CTX_"):
+ forwarded[k] = v
+ if config.env:
+ for k, v in config.env.items():
+ forwarded[k] = v
+
+ for k in (
+ "HTTP_PROXY",
+ "HTTPS_PROXY",
+ "NO_PROXY",
+ "http_proxy",
+ "https_proxy",
+ "no_proxy",
+ ):
+ if k in env:
+ forwarded[k] = env[k]
+
+ if "PYTHONIOENCODING" in env:
+ forwarded["PYTHONIOENCODING"] = env["PYTHONIOENCODING"]
+
+ for k, v in forwarded.items():
+ if "://127.0.0.1:" in v:
+ forwarded[k] = v.replace("://127.0.0.1:", f"://{host_gateway}:")
+ if "://localhost:" in v:
+ forwarded[k] = v.replace("://localhost:", f"://{host_gateway}:")
+
+ # Ensure host_gateway is excluded from proxy so container can reach
+ # host-side services (LiteLLM proxy, MCP server) directly.
+ for k in ("NO_PROXY", "no_proxy"):
+ existing = forwarded.get(k, "")
+ if host_gateway not in existing:
+ forwarded[k] = f"{existing},{host_gateway}" if existing else host_gateway
+
+ return forwarded
+
+ def _patch_mcp_json(self, *, cfg_root: Path, host_gateway: str) -> None:
+ mcp_path = cfg_root / "mcp.json"
+ if not mcp_path.exists():
+ return
+
+ try:
+ data = json.loads(mcp_path.read_text(encoding="utf-8"))
+ env_cfg = data.get("mcpServers", {}).get("environment", {})
+ url = env_cfg.get("url", "")
+ if not isinstance(url, str) or not url:
+ return
+
+ if "://127.0.0.1:" in url:
+ data["mcpServers"]["environment"]["url"] = url.replace("://127.0.0.1:", f"://{host_gateway}:")
+ mcp_path.write_text(json.dumps(data, indent=2), encoding="utf-8")
+ self._logger.debug("Patched mcp.json for container host gateway: %s", mcp_path)
+ elif "://localhost:" in url:
+ data["mcpServers"]["environment"]["url"] = url.replace("://localhost:", f"://{host_gateway}:")
+ mcp_path.write_text(json.dumps(data, indent=2), encoding="utf-8")
+ self._logger.debug("Patched mcp.json for container host gateway: %s", mcp_path)
+
+ except Exception as exc:
+ self._logger.warning("Failed to patch mcp.json (%s): %s", mcp_path, exc)
+
+
+class PodmanRunner(ContainerRunner):
+ """Run the inner command inside a container via Podman."""
+
+ host_gateway = "host.containers.internal"
+
+ def __init__(self, log_path, logger):
+ super().__init__(log_path, logger)
+
+ def run(
+ self,
+ *,
+ cmd: list[str],
+ env: dict[str, str],
+ cfg_root: Path,
+ config: BaseCLIConfig,
+ spawn_error_message: str,
+ ) -> CLIResult:
+ runtime = "podman"
+ host_gateway = "host.containers.internal"
+ host_cfg_root = str(cfg_root.resolve())
+
+ # Patch mcp.json so container uses host gateway (not 127.0.0.1/localhost)
+ self._patch_mcp_json(cfg_root=cfg_root, host_gateway=host_gateway)
+
+ inner_cmd = self._rewrite_mcp_config_path(list(cmd), workdir=config.image_workdir)
+
+ # Minimal env forwarding into container (encoded via podman -e flags)
+ container_env = self._container_env_from(env, host_gateway=host_gateway, config=config)
+ container_env["HOME"] = config.image_workdir
+ connection_args = self._resolve_podman_connection_args()
+ user_args: list[str] = []
+ uid = getattr(os, "getuid", None)
+ gid = getattr(os, "getgid", None)
+ if callable(uid) and callable(gid):
+ try:
+ user_args = ["--user", f"{uid()}:{gid()}"]
+ except Exception:
+ user_args = []
+ wrapped_cmd: list[str] = [
+ runtime,
+ *connection_args,
+ "run",
+ "--rm",
+ *user_args,
+ "-v",
+ f"{host_cfg_root}:{config.image_workdir}:Z",
+ "-w",
+ config.image_workdir,
+ ]
+ for k, v in container_env.items():
+ wrapped_cmd.extend(["-e", f"{k}={v}"])
+ wrapped_cmd.append(str(config.image))
+ wrapped_cmd.extend(inner_cmd)
+
+ # Important: do NOT keep stdin open (avoids "podman run never ends")
+ return super().run(
+ cmd=wrapped_cmd,
+ env=env,
+ cfg_root=cfg_root,
+ config=config,
+ spawn_error_message=spawn_error_message,
+ stdin_devnull=True,
+ )
+
+ def _resolve_podman_connection_args(self) -> list[str]:
+ """Return extra args for the `podman` CLI to select a connection.
+
+ Priority:
+ 1) PODMAN_CONNECTION env var
+ 2) auto-detect default connection from `podman system connection list --format json`
+ 3) fallback: no args (let Podman decide; works on native Linux / preconfigured env)
+ """
+ # 1) environment override (nice for CI/users)
+ env_name = os.environ.get("PODMAN_CONNECTION")
+ if env_name:
+ return ["--connection", env_name]
+
+ # 2) auto-detect default connection
+ try:
+ proc = subprocess.run(
+ ["podman", "system", "connection", "list", "--format", "json"],
+ check=False,
+ capture_output=True,
+ text=True,
+ )
+ if proc.returncode != 0:
+ self._logger.info(
+ "podman system connection list failed (rc=%s): %s",
+ proc.returncode,
+ (proc.stderr or "").strip(),
+ )
+ return []
+
+ data = json.loads(proc.stdout or "[]")
+ # entries look like: {"Name": "...", "URI": "...", "Identity": "...", "Default": true, ...}
+ default = next((x for x in data if x.get("Default") is True), None)
+ if default and default.get("Name"):
+ return [
+ "--url",
+ str(default["URI"]),
+ "--identity",
+ str(default["Identity"]),
+ ]
+ except Exception as exc:
+ self._logger.info("Failed to auto-detect Podman connection: %r", exc)
+ return []
+
+ return []
+
+
+class DockerRunner(ContainerRunner):
+ """Run the inner command inside a container via Docker."""
+
+ def __init__(self, log_path, logger):
+ super().__init__(log_path, logger)
+
+ def run(
+ self,
+ *,
+ cmd: list[str],
+ env: dict[str, str],
+ cfg_root: Path,
+ config: BaseCLIConfig,
+ spawn_error_message: str,
+ ) -> CLIResult:
+ runtime = "docker"
+ host_gateway = "host.docker.internal"
+ host_cfg_root = str(cfg_root.resolve())
+
+ # Patch mcp.json so container uses host gateway (not 127.0.0.1/localhost)
+ self._patch_mcp_json(cfg_root=cfg_root, host_gateway=host_gateway)
+
+ inner_cmd = self._rewrite_mcp_config_path(list(cmd), workdir=config.image_workdir)
+
+ # Minimal env forwarding into container (encoded via docker -e flags)
+ container_env = self._container_env_from(env, host_gateway=host_gateway, config=config)
+ container_env["HOME"] = config.image_workdir
+ user_args: list[str] = []
+ uid = getattr(os, "getuid", None)
+ gid = getattr(os, "getgid", None)
+ if callable(uid) and callable(gid):
+ try:
+ user_args = ["--user", f"{uid()}:{gid()}"]
+ except Exception:
+ user_args = []
+ wrapped_cmd: list[str] = [
+ runtime,
+ "run",
+ "--rm",
+ "--add-host",
+ f"{host_gateway}:host-gateway",
+ *user_args,
+ "-v",
+ f"{host_cfg_root}:{config.image_workdir}",
+ "-w",
+ config.image_workdir,
+ ]
+ for k, v in container_env.items():
+ wrapped_cmd.extend(["-e", f"{k}={v}"])
+ wrapped_cmd.append(str(config.image))
+ wrapped_cmd.extend(inner_cmd)
+
+ # Important: do NOT keep stdin open (avoids "docker run never ends")
+ return super().run(
+ cmd=wrapped_cmd,
+ env=env,
+ cfg_root=cfg_root,
+ config=config,
+ spawn_error_message=spawn_error_message,
+ stdin_devnull=True,
+ )
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/cli/gemini/__init__.py b/labs/AgentStream/exgentic/src/exgentic/agents/cli/gemini/__init__.py
new file mode 100644
index 00000000..f14bde1d
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/cli/gemini/__init__.py
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from .agent import GeminiAgent, GeminiAgentInstance # noqa: F401
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/cli/gemini/agent.py b/labs/AgentStream/exgentic/src/exgentic/agents/cli/gemini/agent.py
new file mode 100644
index 00000000..19067ccf
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/cli/gemini/agent.py
@@ -0,0 +1,81 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from __future__ import annotations
+
+import os
+from typing import Any, ClassVar
+
+from ....core.types import ModelSettings
+from ..base import ExecutionBackend, ProxyBackedAgent, ProxyBackedMCPAgentInstance
+from .cli import GeminiCLI, GeminiCLIConfig
+
+
+class GeminiAgentInstance(ProxyBackedMCPAgentInstance):
+ """Self-contained Gemini CLI agent that runs through a LiteLLM proxy."""
+
+ def __init__(
+ self,
+ session_id: str,
+ model_id: str,
+ max_steps: int = 150,
+ execution_backend: ExecutionBackend = ExecutionBackend.AUTO,
+ model_settings: ModelSettings | None = None,
+ ):
+ self._gemini_model_alias = "gemini-2.5-pro"
+ super().__init__(
+ session_id,
+ model_id,
+ max_steps=max_steps,
+ model_alias=self._gemini_model_alias,
+ execution_backend=execution_backend,
+ model_settings=model_settings,
+ )
+ self._gemini_log = self.paths.agent_dir / "gemini_cli.log"
+
+ @property
+ def cli_display_name(self) -> str:
+ return "Gemini CLI"
+
+ def _build_cli(self) -> GeminiCLI:
+ cfg_dir = self.paths.agent_dir / "gemini_config"
+ return GeminiCLI(
+ env=os.environ.copy(),
+ log_path=self._gemini_log,
+ config_dir=cfg_dir,
+ logger=self.logger,
+ runner=self.execution_backend,
+ )
+
+ def _run_cli(
+ self,
+ cli: GeminiCLI,
+ prompt: str,
+ mcp_host: str,
+ mcp_port: int,
+ proxy: Any,
+ ) -> Any:
+ config = GeminiCLIConfig(
+ mcp_host=mcp_host,
+ mcp_port=mcp_port,
+ provider_url=proxy.base_url,
+ backend_model=self.model_id,
+ gemini_model=self._gemini_model_alias,
+ allowed_mcp_server_names=["environment"],
+ allowed_tools=[action.name for action in self.actions],
+ )
+ result = cli.run(prompt=prompt, config=config)
+ return result.stdout
+
+
+class GeminiAgent(ProxyBackedAgent):
+ display_name: ClassVar[str] = "Gemini CLI"
+ slug_name: ClassVar[str] = "gemini_cli"
+ execution_backend: ExecutionBackend = ExecutionBackend.AUTO
+
+ @classmethod
+ def _get_instance_class(cls):
+ return GeminiAgentInstance
+
+ def get_models_names(self) -> list[str]: # type: ignore[override]
+ return [str(self.model_id)]
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/cli/gemini/cli.py b/labs/AgentStream/exgentic/src/exgentic/agents/cli/gemini/cli.py
new file mode 100644
index 00000000..935d3fa7
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/cli/gemini/cli.py
@@ -0,0 +1,112 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Optional
+
+from ..base import BaseCLIConfig, BaseCLIWrapper, ExecutionBackend
+
+
+class GeminiCLIConfig(BaseCLIConfig):
+ backend_model: str
+ gemini_model: str = "gemini-2.5-pro"
+ env_key: str = "GEMINI_API_KEY"
+ server_name: str = "environment"
+ output_format: str = "text"
+ approval_mode: str = "yolo"
+ allowed_mcp_server_names: Optional[list[str]] = None
+ allowed_tools: Optional[list[str]] = None
+ image: str = "exgentic-gemini:dev"
+
+
+class GeminiCLI(BaseCLIWrapper):
+ """Lightweight wrapper for running the Gemini CLI headlessly."""
+
+ def __init__(
+ self,
+ env: Optional[dict[str, str]] = None,
+ log_path: Optional[Path] = None,
+ config_dir: Optional[Path] = None,
+ logger=None,
+ runner: ExecutionBackend = ExecutionBackend.PROCESS,
+ ) -> None:
+ super().__init__(
+ env=env,
+ log_path=log_path,
+ config_dir=config_dir,
+ logger=logger,
+ runner=runner,
+ )
+ self._settings_path: Optional[Path] = None
+ self.config_prefix = "gemini_cli_"
+ self.spawn_error_message = "Failed to start Gemini CLI"
+
+ # Required hooks --------------------------------------------------
+
+ def build_env(self, *, cfg_root: Path, prompt: str, config: GeminiCLIConfig) -> dict[str, str]:
+ env = self.env.copy()
+ env["GOOGLE_GEMINI_BASE_URL"] = config.provider_url
+
+ api_key = env.get(config.env_key) or env.get("GEMINI_API_KEY") or "dummy-api-key"
+ env[config.env_key] = api_key
+ env["GEMINI_API_KEY"] = api_key
+
+ env["HOME"] = str(cfg_root)
+ return env
+
+ def build_command(self, *, cfg_root: Path, prompt: str, config: GeminiCLIConfig) -> list[str]:
+ gemini_cfg_dir = cfg_root / ".gemini"
+ settings_path = gemini_cfg_dir / "settings.json"
+ self._settings_path = settings_path
+
+ # Rewrite localhost addresses to host gateway for container runners
+ mcp_host = config.mcp_host
+ from ..command_runner import ContainerRunner
+
+ if isinstance(self.runner, ContainerRunner) and mcp_host in ("0.0.0.0", "127.0.0.1", "localhost"):
+ mcp_host = self.runner.host_gateway
+
+ gemini_cfg_dir.mkdir(parents=True, exist_ok=True)
+ mcp_url = f"http://{mcp_host}:{config.mcp_port}/mcp"
+ self._ensure_settings(settings_path, config.server_name, mcp_url)
+
+ cmd: list[str] = [
+ "gemini",
+ "--model",
+ config.gemini_model,
+ "--output-format",
+ config.output_format,
+ "--approval-mode",
+ config.approval_mode,
+ ]
+ if config.allowed_mcp_server_names:
+ cmd.extend(["--allowed-mcp-server-names", *config.allowed_mcp_server_names])
+ if config.allowed_tools:
+ cmd.extend(["--allowed-tools", *config.allowed_tools])
+ cmd.append(prompt)
+ return cmd
+
+ # Internal helpers -------------------------------------------------
+
+ def _ensure_settings(self, settings_path: Path, server_name: str, mcp_url: str) -> None:
+ settings: dict[str, object] = {}
+ if settings_path.exists():
+ try:
+ with open(settings_path, encoding="utf-8-sig") as fh:
+ settings = json.load(fh)
+ except Exception:
+ settings = {}
+
+ servers = settings.setdefault("mcpServers", {})
+ if not isinstance(servers, dict):
+ servers = {}
+ settings["mcpServers"] = servers
+
+ servers[server_name] = {"httpUrl": mcp_url, "trust": True}
+
+ settings_path.parent.mkdir(parents=True, exist_ok=True)
+ with open(settings_path, "w", encoding="utf-8") as fh:
+ json.dump(settings, fh, indent=2)
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/cli/gemini/setup.sh b/labs/AgentStream/exgentic/src/exgentic/agents/cli/gemini/setup.sh
new file mode 100644
index 00000000..f279d55d
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/cli/gemini/setup.sh
@@ -0,0 +1,32 @@
+#!/bin/bash
+set -euo pipefail
+
+# Determine container runtime
+CONTAINER_CMD=""
+if command -v podman >/dev/null 2>&1; then
+ CONTAINER_CMD="podman"
+ # Start podman machine if needed (macOS/Windows)
+ if podman machine list >/dev/null 2>&1; then
+ MACHINE_STATUS=$(podman machine list --format "{{.Running}}" 2>/dev/null | head -n 1)
+ if [ -z "$MACHINE_STATUS" ]; then
+ podman machine init && podman machine start
+ elif [ "$MACHINE_STATUS" != "true" ]; then
+ podman machine start
+ fi
+ fi
+elif command -v docker >/dev/null 2>&1; then
+ CONTAINER_CMD="docker"
+else
+ echo "Error: Neither Podman nor Docker found." >&2
+ exit 1
+fi
+
+# Build Gemini CLI container image (inline — no external Dockerfile needed)
+$CONTAINER_CMD build -t exgentic-gemini:dev -f - . <<'DOCKERFILE'
+FROM registry.access.redhat.com/ubi9/nodejs-20
+RUN npm install -g @google/gemini-cli@0.25.0
+WORKDIR /work
+CMD ["gemini","--help"]
+DOCKERFILE
+
+echo "Gemini Agent setup complete (using $CONTAINER_CMD)"
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/cli/requirements.txt b/labs/AgentStream/exgentic/src/exgentic/agents/cli/requirements.txt
new file mode 100644
index 00000000..57fa7d59
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/cli/requirements.txt
@@ -0,0 +1 @@
+litellm[proxy]
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/harness/__init__.py b/labs/AgentStream/exgentic/src/exgentic/agents/harness/__init__.py
new file mode 100644
index 00000000..e096de68
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/harness/__init__.py
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from .harness_agent import HarnessAgent
+
+__all__ = ["HarnessAgent"]
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/harness/evolver.py b/labs/AgentStream/exgentic/src/exgentic/agents/harness/evolver.py
new file mode 100644
index 00000000..5dea5038
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/harness/evolver.py
@@ -0,0 +1,382 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import json
+import logging
+import time
+from typing import Any, Dict, List
+
+import litellm
+
+from .harness_store import HarnessStore
+from .prompts.evolver import EVOLVER_SYSTEM_PROMPT, build_evolution_user_message
+from .retriever import compute_embedding
+
+logger = logging.getLogger(__name__)
+
+EVOLVER_TOOLS = [
+ {
+ "type": "function",
+ "function": {
+ "name": "read_prompt",
+ "description": "Read the current system prompt.",
+ "parameters": {"type": "object", "properties": {}, "required": []},
+ },
+ },
+ {
+ "type": "function",
+ "function": {
+ "name": "read_memory",
+ "description": "Read the current long-term memory document.",
+ "parameters": {"type": "object", "properties": {}, "required": []},
+ },
+ },
+ {
+ "type": "function",
+ "function": {
+ "name": "list_skills",
+ "description": "List all skills with their names and descriptions.",
+ "parameters": {"type": "object", "properties": {}, "required": []},
+ },
+ },
+ {
+ "type": "function",
+ "function": {
+ "name": "read_skill",
+ "description": "Read the full body of a specific skill.",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "name": {"type": "string", "description": "The skill name to read."},
+ },
+ "required": ["name"],
+ },
+ },
+ },
+ {
+ "type": "function",
+ "function": {
+ "name": "edit_prompt",
+ "description": "Replace the entire system prompt with new content.",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "body": {"type": "string", "description": "The full new system prompt text."},
+ },
+ "required": ["body"],
+ },
+ },
+ },
+ {
+ "type": "function",
+ "function": {
+ "name": "edit_memory",
+ "description": "Replace the entire long-term memory document with new content.",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "body": {"type": "string", "description": "The full new memory document text."},
+ },
+ "required": ["body"],
+ },
+ },
+ },
+ {
+ "type": "function",
+ "function": {
+ "name": "add_skill",
+ "description": "Add a new skill to the skill library.",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "name": {"type": "string", "description": "Unique skill name."},
+ "description": {"type": "string", "description": "One-line description of what the skill does."},
+ "body": {"type": "string", "description": "Full skill content/instructions."},
+ },
+ "required": ["name", "description", "body"],
+ },
+ },
+ },
+ {
+ "type": "function",
+ "function": {
+ "name": "edit_skill",
+ "description": "Modify an existing skill's description and/or body.",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "name": {"type": "string", "description": "The skill name to edit."},
+ "description": {"type": "string", "description": "New description (optional, omit to keep current)."},
+ "body": {"type": "string", "description": "New body (optional, omit to keep current)."},
+ },
+ "required": ["name"],
+ },
+ },
+ },
+ {
+ "type": "function",
+ "function": {
+ "name": "delete_skill",
+ "description": "Delete a skill from the skill library.",
+ "parameters": {
+ "type": "object",
+ "properties": {
+ "name": {"type": "string", "description": "The skill name to delete."},
+ },
+ "required": ["name"],
+ },
+ },
+ },
+]
+
+
+# ================================================================== #
+# Tool executor
+# ================================================================== #
+
+class _EvolverToolExecutor:
+ """Executes evolver tools against a HarnessStore.
+
+ Tracks changes and enforces constraints (at most 1 edit_prompt, 1 edit_memory).
+ """
+
+ def __init__(self, store: HarnessStore, embedding_model: str) -> None:
+ self.store = store
+ self.embedding_model = embedding_model
+ self.ops_applied: List[str] = []
+ self._edit_prompt_used = False
+ self._edit_memory_used = False
+
+ def execute(self, tool_name: str, args: Dict[str, Any]) -> str:
+ """Execute a tool call and return the result string."""
+ if tool_name == "read_prompt":
+ return self.store.system_prompt or "(Empty)"
+
+ elif tool_name == "read_memory":
+ return self.store.memory or "(Empty)"
+
+ elif tool_name == "list_skills":
+ index = self.store.get_skill_index()
+ if not index:
+ return "(No skills yet)"
+ return json.dumps(index, ensure_ascii=False, indent=2)
+
+ elif tool_name == "read_skill":
+ name = args.get("name", "")
+ skills = {s.name: s for s in self.store.list_skills()}
+ skill = skills.get(name)
+ if skill is None:
+ return f"ERROR: Skill '{name}' not found."
+ return f"Name: {skill.name}\nDescription: {skill.description}\n\n{skill.body}"
+
+ elif tool_name == "edit_prompt":
+ if self._edit_prompt_used:
+ return "ERROR: edit_prompt already used this session (limit: 1 per task)."
+ body = args.get("body", "")
+ if not body:
+ return "ERROR: 'body' is required."
+ self.store.edit_prompt(body)
+ self._edit_prompt_used = True
+ self.ops_applied.append("edit_prompt")
+ return "OK: System prompt updated."
+
+ elif tool_name == "edit_memory":
+ if self._edit_memory_used:
+ return "ERROR: edit_memory already used this session (limit: 1 per task)."
+ body = args.get("body", "")
+ self.store.edit_memory(body)
+ self._edit_memory_used = True
+ self.ops_applied.append("edit_memory")
+ return "OK: Memory updated."
+
+ elif tool_name == "add_skill":
+ name = args.get("name", "")
+ description = args.get("description", "")
+ body = args.get("body", "")
+ if not name or not description or not body:
+ return "ERROR: 'name', 'description', and 'body' are all required."
+ # Check if skill already exists
+ existing = {s.name for s in self.store.list_skills()}
+ if name in existing:
+ return f"ERROR: Skill '{name}' already exists. Use edit_skill to modify it."
+ self.store.add_skill(name, description, body)
+ # Compute embedding
+ emb = compute_embedding(description, model=self.embedding_model)
+ if emb:
+ self.store.set_embedding(name, emb)
+ self.ops_applied.append(f"add_skill:{name}")
+ return f"OK: Skill '{name}' added."
+
+ elif tool_name == "edit_skill":
+ name = args.get("name", "")
+ if not name:
+ return "ERROR: 'name' is required."
+ description = args.get("description")
+ body = args.get("body")
+ if description is None and body is None:
+ return "ERROR: At least one of 'description' or 'body' must be provided."
+ success = self.store.edit_skill(name, description=description, body=body)
+ if not success:
+ return f"ERROR: Skill '{name}' not found."
+ if description:
+ emb = compute_embedding(description, model=self.embedding_model)
+ if emb:
+ self.store.set_embedding(name, emb)
+ self.ops_applied.append(f"edit_skill:{name}")
+ return f"OK: Skill '{name}' updated."
+
+ elif tool_name == "delete_skill":
+ name = args.get("name", "")
+ if not name:
+ return "ERROR: 'name' is required."
+ success = self.store.delete_skill(name)
+ if not success:
+ return f"ERROR: Skill '{name}' not found."
+ self.ops_applied.append(f"delete_skill:{name}")
+ return f"OK: Skill '{name}' deleted."
+
+ else:
+ return f"ERROR: Unknown tool '{tool_name}'."
+
+
+# ================================================================== #
+# Multi-turn evolver loop
+# ================================================================== #
+
+def _run_evolver_loop(
+ model: str,
+ system_prompt: str,
+ user_message: str,
+ executor: _EvolverToolExecutor,
+ max_turns: int = 20,
+) -> None:
+ """Run multi-turn evolver with tools.
+
+ The LLM can read harness state, then make changes via tool calls.
+ Loop ends when LLM stops calling tools (finish_reason != tool_calls).
+ """
+ messages: List[Dict[str, Any]] = [
+ {"role": "system", "content": system_prompt},
+ {"role": "user", "content": user_message},
+ ]
+
+ for turn in range(max_turns):
+ # Call LLM
+ max_attempts = 3
+ response = None
+ for attempt in range(max_attempts):
+ try:
+ response = litellm.completion(
+ model=model,
+ messages=messages,
+ tools=EVOLVER_TOOLS,
+ temperature=0.0,
+ )
+ break
+ except Exception as exc:
+ logger.warning("Evolver loop turn %d attempt %d failed: %s", turn, attempt + 1, exc)
+ if attempt + 1 >= max_attempts:
+ logger.error("Evolver loop: all attempts failed at turn %d", turn)
+ return
+ time.sleep(2 ** attempt)
+
+ if response is None:
+ break
+
+ choice = response.choices[0]
+ message = choice.message
+
+ # Check for tool calls
+ if hasattr(message, "tool_calls") and message.tool_calls:
+ # Add assistant message to history
+ assistant_msg: Dict[str, Any] = {"role": "assistant", "content": message.content or ""}
+ assistant_msg["tool_calls"] = [
+ {
+ "id": tc.id,
+ "type": "function",
+ "function": {
+ "name": tc.function.name,
+ "arguments": tc.function.arguments,
+ },
+ }
+ for tc in message.tool_calls
+ ]
+ messages.append(assistant_msg)
+
+ # Execute each tool call
+ for tc in message.tool_calls:
+ try:
+ args = json.loads(tc.function.arguments) if tc.function.arguments else {}
+ except json.JSONDecodeError:
+ args = {}
+
+ result = executor.execute(tc.function.name, args)
+ # Log each tool call with args summary and result preview
+ args_summary = ", ".join(f"{k}={repr(v)[:60]}" for k, v in args.items())
+ logger.info(
+ "Evolver turn %d: %s(%s) → %s",
+ turn, tc.function.name, args_summary, result[:120],
+ )
+ messages.append({
+ "role": "tool",
+ "tool_call_id": tc.id,
+ "content": result,
+ })
+
+ continue # Next turn
+ else:
+ # LLM finished (no more tool calls)
+ logger.info("Evolver finished after %d turns", turn + 1)
+ break
+
+
+# ================================================================== #
+# Main evolver entry point
+# ================================================================== #
+
+def run_evolver(
+ store: HarnessStore,
+ task: str,
+ injected_skill_names: List[str],
+ trajectory: str,
+ llm_call: Any, # kept for interface compat (unused in multi-turn impl)
+ evolver_model: str,
+ embedding_model: str = "all-MiniLM-L6-v2",
+) -> tuple[int, List[str]]:
+ """Run the evolver: multi-turn tool-calling to read and modify harness.
+
+ Returns (number of ops applied, list of op summaries).
+ """
+ # Take snapshot for full rollback on catastrophic failure
+ full_snapshot = store.snapshot()
+
+ try:
+ # Build user message
+ user_message = build_evolution_user_message(
+ task=task,
+ injected_skill_names=injected_skill_names,
+ trajectory=trajectory,
+ )
+
+ # Create tool executor
+ executor = _EvolverToolExecutor(store, embedding_model)
+
+ # Run multi-turn loop
+ _run_evolver_loop(
+ model=evolver_model,
+ system_prompt=EVOLVER_SYSTEM_PROMPT,
+ user_message=user_message,
+ executor=executor,
+ max_turns=20,
+ )
+
+ ops_applied = len(executor.ops_applied)
+ logger.info("Harness evolver: %d ops applied: %s", ops_applied, executor.ops_applied)
+ return ops_applied, executor.ops_applied
+
+ except Exception as exc:
+ logger.warning("Harness evolver failed, rolling back: %s", exc)
+ store.rollback(full_snapshot)
+ return 0, []
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/harness/harness_agent.py b/labs/AgentStream/exgentic/src/exgentic/agents/harness/harness_agent.py
new file mode 100644
index 00000000..19dcc5db
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/harness/harness_agent.py
@@ -0,0 +1,70 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+from typing import Any, ClassVar, Optional
+
+from pydantic import ConfigDict
+
+from ...core.agent import Agent
+from ...core.types import ModelSettings
+from ...utils.settings import RunnerName
+
+
+class HarnessAgent(Agent):
+
+ display_name: ClassVar[str] = "Harness Agent"
+ slug_name: ClassVar[str] = "harness"
+
+ model_config = ConfigDict(arbitrary_types_allowed=True)
+
+ model: str = "gpt-4o"
+ evolver_model: Optional[str] = None
+
+ top_k_skills: int = 3
+ embedding_model: str = "all-MiniLM-L6-v2"
+
+ shuffle_mode: str = "isolated"
+
+ benchmark_id: Optional[str] = None
+
+ enable_tool_shortlisting: bool = False
+ max_selected_tools: int = 30
+
+ runner: RunnerName | None = None
+ model_settings: ModelSettings | None = None
+
+ @classmethod
+ def _get_instance_class(cls):
+ from .harness_instance import HarnessAgentInstance
+ return HarnessAgentInstance
+
+ @classmethod
+ def _get_instance_class_ref(cls) -> str:
+ return "exgentic.agents.harness.harness_instance:HarnessAgentInstance"
+
+ def _get_instance_kwargs(self, session_id: str) -> dict[str, Any]:
+ return {
+ "session_id": session_id,
+ "model": self.model,
+ "evolver_model": self.evolver_model or self.model,
+ "top_k_skills": self.top_k_skills,
+ "embedding_model": self.embedding_model,
+ "shuffle_mode": self.shuffle_mode,
+ "model_settings": self.model_settings,
+ "benchmark_id": self.benchmark_id,
+ "enable_tool_shortlisting": self.enable_tool_shortlisting,
+ "max_selected_tools": self.max_selected_tools,
+ }
+
+ @property
+ def model_name(self) -> str:
+ return str(self.model).split("/")[-1]
+
+ def get_models_names(self) -> list[str]:
+ names = [str(self.model)]
+ em = self.evolver_model or self.model
+ if em != self.model:
+ names.append(str(em))
+ return names
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/harness/harness_instance.py b/labs/AgentStream/exgentic/src/exgentic/agents/harness/harness_instance.py
new file mode 100644
index 00000000..14fc1d27
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/harness/harness_instance.py
@@ -0,0 +1,579 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import json
+import time
+from datetime import datetime
+from typing import Any, Dict, List, Optional, Union
+
+import litellm
+from litellm import (
+ ChatCompletionAssistantMessage,
+ ChatCompletionSystemMessage,
+ ChatCompletionToolMessage,
+ ChatCompletionUserMessage,
+)
+
+from ...core.agent_instance import AgentInstance
+from ...core.types import (
+ Action,
+ ActionType,
+ Message,
+ MessageAction,
+ MessageObservation,
+ MessagePayload,
+ ModelSettings,
+ Observation,
+)
+from ...utils.cost import LiteLLMCostReport
+from ...utils.settings import get_settings
+
+from .evolver import run_evolver
+from .harness_store import HarnessStore
+from .prompts.inject import build_system_message
+from .retriever import retrieve_skills
+
+try:
+ from ...agents.litellm_tool_calling.utils import ToolCall, ToolsActionsRegistry
+except ImportError:
+ ToolsActionsRegistry = None
+ ToolCall = dict
+
+from ..tool_shortlisting import shortlist_tools
+
+settings = get_settings()
+
+
+class HarnessAgentInstance(AgentInstance):
+
+ def __init__(
+ self,
+ session_id: str,
+ model: str = "gpt-4o",
+ evolver_model: str = "gpt-4o",
+ top_k_skills: int = 3,
+ embedding_model: str = "all-MiniLM-L6-v2",
+ shuffle_mode: str = "isolated",
+ model_settings: Optional[ModelSettings] = None,
+ benchmark_id: Optional[str] = None,
+ enable_tool_shortlisting: bool = False,
+ max_selected_tools: int = 30,
+ ) -> None:
+ super().__init__(session_id)
+
+ self.model = model
+ self.evolver_model = evolver_model
+ self.top_k_skills = top_k_skills
+ self.embedding_model = embedding_model
+ self.shuffle_mode = shuffle_mode
+ self.benchmark_id = benchmark_id
+ self.enable_tool_shortlisting = enable_tool_shortlisting
+ self.max_selected_tools = max_selected_tools
+
+ if model_settings is None:
+ self._model_settings = ModelSettings()
+ elif isinstance(model_settings, ModelSettings):
+ self._model_settings = model_settings
+ else:
+ self._model_settings = ModelSettings()
+
+ self._cost = LiteLLMCostReport.initialize_empty(model_name=self.model)
+ self._store: Optional[HarnessStore] = None
+
+ self.messages: list[
+ Union[
+ ChatCompletionAssistantMessage,
+ ChatCompletionToolMessage,
+ ChatCompletionUserMessage,
+ ChatCompletionSystemMessage,
+ ]
+ ] = []
+ self._step_count: int = 0
+
+ self._registry: Optional[ToolsActionsRegistry] = None
+ self._all_actions: list[ActionType] = []
+
+ self._observation_log: List[Dict[str, Any]] = []
+ self._action_log: List[Dict[str, Any]] = []
+
+ self._injected_skill_names: List[str] = []
+
+ def _log_failure(
+ self, component: str, error: Exception, context: Dict[str, Any]
+ ) -> None:
+ try:
+ log_path = self.paths.agent_dir / "harness_failures.jsonl"
+ log_path.parent.mkdir(parents=True, exist_ok=True)
+ entry = {
+ "timestamp": datetime.now().isoformat(),
+ "session_id": self.session_id,
+ "component": component,
+ "error_type": type(error).__name__,
+ "error_message": str(error)[:2000],
+ **{k: str(v)[:2000] if isinstance(v, str) else v
+ for k, v in context.items()},
+ }
+ with open(log_path, "a", encoding="utf-8") as fh:
+ fh.write(json.dumps(entry, ensure_ascii=False) + "\n")
+ except Exception:
+ pass
+
+ def start(
+ self,
+ task: str,
+ context: Dict[str, Any],
+ actions: list[ActionType],
+ ) -> None:
+ super().start(task, context, actions)
+
+ self._all_actions = list(self.actions)
+ if ToolsActionsRegistry is not None:
+ self._registry = ToolsActionsRegistry(self._all_actions)
+
+ task_group = str(
+ context.get("task_group")
+ or context.get("task_id")
+ or context.get("task_name")
+ or "default"
+ )
+ self._store = HarnessStore.get_or_create(
+ shuffle_mode=self.shuffle_mode,
+ task_group=task_group,
+ benchmark_id=self.benchmark_id,
+ )
+ self._store.increment_session()
+
+ retrieved = []
+ if self._store.skill_count > 0:
+ try:
+ retrieved = retrieve_skills(
+ task_text=task,
+ store=self._store,
+ top_k=self.top_k_skills,
+ embedding_model=self.embedding_model,
+ )
+ except Exception as exc:
+ self.logger.warning("Harness: skill retrieval failed: %s", exc)
+
+ self._injected_skill_names = [s.name for s, _ in retrieved]
+
+ # Mark retrieved skills as used (LRU)
+ if self._injected_skill_names:
+ self._store.touch_skills(self._injected_skill_names)
+
+ # Build system message
+ system_content = build_system_message(
+ system_prompt=self._store.system_prompt,
+ memory=self._store.memory,
+ retrieved_skills=retrieved,
+ )
+ self._add_message(
+ ChatCompletionSystemMessage(role="system", content=system_content)
+ )
+ self.logger.info(
+ "Harness system message built: prompt_len=%d memory_len=%d "
+ "skills_injected=%s total_system_len=%d",
+ len(self._store.system_prompt),
+ len(self._store.memory),
+ self._injected_skill_names,
+ len(system_content),
+ )
+
+ content_parts: list[Any] = []
+ ctx = ""
+ if self.context:
+ for k, v in self.context.items():
+ if isinstance(v, dict) and v.get("type") == "image_url":
+ content_parts.append({"type": "image_url", "image_url": {"url": v["data"], "detail": "high"}})
+ else:
+ ctx += f"\n<{k}>\n{v}\n"
+
+ text_content = f"{self.task}\n{ctx}"
+ if content_parts:
+ content_parts.insert(0, {"type": "text", "text": text_content})
+ self._add_message(ChatCompletionUserMessage(role="user", content=content_parts))
+ else:
+ self._add_message(
+ ChatCompletionUserMessage(role="user", content=text_content)
+ )
+
+ self.logger.info(
+ "Harness instance started store=%s session_count=%d "
+ "skill_count=%d skills_injected=%d benchmark=%s tools=%d",
+ self._store.store_id,
+ self._store.session_count,
+ self._store.skill_count,
+ len(self._injected_skill_names),
+ self.benchmark_id or "(none)",
+ len(self._all_actions),
+ )
+
+ def react(self, observation: Optional[Observation]) -> Optional[Action]:
+ self._step_count += 1
+
+ self._observe(observation)
+ self._log_observation(observation)
+
+ tools = self._assistant_tools()
+ response = self._completion(
+ model=self.model,
+ messages=self.messages,
+ tools=tools if tools else None,
+ )
+
+ if response is None:
+ self.logger.error("Harness: LLM returned None response")
+ return None
+
+ if response.usage:
+ self._cost.update_cost_from_tokens(
+ response.usage.prompt_tokens,
+ response.usage.completion_tokens,
+ )
+
+ choice = response["choices"][0]
+ message = choice["message"]
+ finish_reason = choice.get("finish_reason")
+
+ if finish_reason == "tool_calls" and self._registry is not None:
+ tool_calls = self._extract_tool_calls(message)
+ self._add_message(
+ ChatCompletionAssistantMessage(
+ role="assistant",
+ tool_calls=[
+ {
+ "id": tc["id"],
+ "type": "function",
+ "function": {
+ "name": tc["name"],
+ "arguments": tc["arguments"],
+ },
+ }
+ for tc in tool_calls
+ ],
+ )
+ )
+ actions = self._registry.tool_calls_to_action(tool_calls)
+
+ for tc in tool_calls:
+ self._action_log.append({
+ "step": self._step_count,
+ "action": tc["name"],
+ "arguments": tc["arguments"],
+ })
+
+ self.logger.info("Harness step %d: tool_calls=%s", self._step_count,
+ [tc["name"] for tc in tool_calls])
+ return actions
+ else:
+ content = message.content if message.content else ""
+ self._add_message(
+ ChatCompletionAssistantMessage(
+ role="assistant", content=content
+ )
+ )
+ self._action_log.append({
+ "step": self._step_count,
+ "action": "message",
+ "content": content,
+ })
+ self.logger.info("Harness step %d: message response", self._step_count)
+ return MessageAction(arguments=Message(content=content))
+
+ def close(self) -> None:
+ store = self._store
+ if store is None:
+ return
+
+ ops_applied = 0
+ ops_summary: list[str] = []
+
+ self.logger.info(
+ "Harness close: starting evolver trajectory_steps=%d "
+ "actions=%d observations=%d",
+ self._step_count, len(self._action_log), len(self._observation_log),
+ )
+
+ if self._observation_log or self._action_log:
+ try:
+ trajectory = self._build_session_trace()
+ self.logger.info(
+ "Harness close: trajectory built length=%d chars", len(trajectory)
+ )
+ ops_applied, ops_summary = run_evolver(
+ store=store,
+ task=self.task if self.task else "",
+ injected_skill_names=self._injected_skill_names,
+ trajectory=trajectory,
+ llm_call=self._llm_call_simple,
+ evolver_model=self.evolver_model,
+ embedding_model=self.embedding_model,
+ )
+ self.logger.info(
+ "Harness close: evolver done ops_applied=%d ops=%s",
+ ops_applied, ops_summary,
+ )
+ except Exception as exc:
+ self.logger.warning("Harness: evolver failed: %s", exc)
+ self._log_failure("evolver", exc, {
+ "observation_count": len(self._observation_log),
+ "action_count": len(self._action_log),
+ })
+
+ if ops_applied > 0:
+ version = store.commit_version(
+ session_id=self.session_id,
+ ops_summary=ops_summary,
+ )
+ self.logger.info(
+ "Harness close: version committed v=%d skill_count=%d",
+ version, store.skill_count,
+ )
+
+ store.record_learning(
+ session_id=self.session_id,
+ task_id=str(self.context.get("task_id", "") if self.context else ""),
+ benchmark_id=self.benchmark_id or "",
+ ops_applied=ops_applied,
+ ops_summary=ops_summary,
+ )
+
+ try:
+ cp = str(self.paths.agent_dir / "harness_checkpoint.json")
+ store.save_checkpoint(cp)
+ txt = str(self.paths.agent_dir / "harness_state.md")
+ store.save_harness_text(txt)
+ except Exception as exc:
+ self.logger.warning("Harness: failed to save checkpoint: %s", exc)
+
+ def get_cost(self) -> LiteLLMCostReport:
+ return self._cost
+
+ def _add_message(self, message: Any) -> None:
+ self.messages.append(message)
+
+ def _observe(self, observation: Optional[Observation]) -> None:
+ if observation is None:
+ return
+
+ observations = observation.to_observation_list()
+ if observation.is_empty():
+ if not any(obs.invoking_actions for obs in observations):
+ return
+
+ for obs in observations:
+ if isinstance(obs, MessageObservation) and isinstance(
+ obs.result, MessagePayload
+ ):
+ self._add_message(
+ ChatCompletionUserMessage(
+ role="user", content=obs.result.message
+ )
+ )
+ continue
+
+ if len(obs.invoking_actions) > 0:
+ invoking = obs.invoking_actions[0]
+ if invoking.name == "message":
+ self._add_message(
+ ChatCompletionUserMessage(
+ role="user", content=str(obs)
+ )
+ )
+ continue
+
+ action_id = invoking.id
+ tool_call_id = invoking.id
+ if not (
+ isinstance(tool_call_id, str)
+ and tool_call_id.startswith("call_")
+ ):
+ if self._registry is not None:
+ tool_call_id = (
+ self._registry.action_id_to_tool_call_id.get(
+ action_id, tool_call_id
+ )
+ )
+
+ value = obs.result
+ try:
+ content = json.dumps(
+ value, ensure_ascii=False, separators=(",", ":")
+ )
+ except TypeError:
+ content = str(value)
+
+ if tool_call_id is not None:
+ self._add_message(
+ ChatCompletionToolMessage(
+ role="tool",
+ tool_call_id=tool_call_id,
+ content=content,
+ )
+ )
+ else:
+ self._add_message(
+ ChatCompletionUserMessage(
+ role="user",
+ content=f"Tool result: {content}",
+ )
+ )
+ else:
+ self._add_message(
+ ChatCompletionUserMessage(
+ role="user", content=str(obs)
+ )
+ )
+
+ def _log_observation(self, observation: Optional[Observation]) -> None:
+ if observation is None or observation.is_empty():
+ return
+
+ for obs in observation.to_observation_list():
+ result = obs.result
+ if result is None:
+ continue
+
+ entry: Dict[str, Any] = {"step": self._step_count}
+ if isinstance(result, str):
+ entry["content"] = result
+ elif isinstance(result, dict):
+ entry["content"] = json.dumps(result, ensure_ascii=False)
+ else:
+ entry["content"] = str(result)
+
+ if obs.invoking_actions:
+ entry["action"] = obs.invoking_actions[0].name
+
+ self._observation_log.append(entry)
+
+ def _assistant_tools(self) -> list[dict[str, Any]]:
+ if self._registry is None:
+ return []
+ tools = self._registry.openai_tools()
+ if not self.enable_tool_shortlisting:
+ return tools
+
+ def _cost_cb(usage):
+ if usage:
+ self._cost.update_cost_from_tokens(
+ usage.prompt_tokens, usage.completion_tokens
+ )
+
+ return shortlist_tools(
+ tools=tools,
+ max_selected=self.max_selected_tools,
+ messages=self.messages,
+ completion_fn=self._completion,
+ model=self.model,
+ logger=self.logger,
+ cost_callback=_cost_cb,
+ )
+
+ @staticmethod
+ def _extract_tool_calls(message: Any) -> list[dict[str, str]]:
+ if not hasattr(message, "tool_calls") or not message.tool_calls:
+ return []
+ tool_calls = []
+ for tc in message.tool_calls:
+ tool_calls.append({
+ "name": tc.function.name,
+ "arguments": tc.function.arguments,
+ "id": tc.id,
+ })
+ return tool_calls
+
+ def _completion(self, **kwargs) -> Any:
+ call_kwargs = self._model_settings.model_dump(
+ exclude_none=True,
+ exclude={"num_retries", "retry_after", "retry_strategy"},
+ )
+ call_kwargs.update(kwargs)
+ if call_kwargs.get("tools") is None:
+ call_kwargs.pop("tools", None)
+
+ max_attempts = 3
+ for attempt in range(max_attempts):
+ try:
+ response = litellm.completion(**call_kwargs)
+ return response
+ except Exception as exc:
+ self.logger.warning(
+ "Harness LLM call attempt %d/%d failed: %s",
+ attempt + 1, max_attempts, exc,
+ )
+ if attempt + 1 >= max_attempts:
+ raise
+ time.sleep(2 ** attempt)
+ return None
+
+ def _llm_call_simple(
+ self,
+ model: str,
+ prompt: str,
+ *,
+ json_mode: bool = False,
+ ) -> str:
+ kwargs: Dict[str, Any] = {
+ "model": model,
+ "messages": [{"role": "user", "content": prompt}],
+ "temperature": 0.0,
+ }
+ if json_mode:
+ kwargs["response_format"] = {"type": "json_object"}
+
+ max_attempts = 3
+ for attempt in range(max_attempts):
+ try:
+ resp = litellm.completion(**kwargs)
+ if resp.usage:
+ self._cost.update_cost_from_tokens(
+ resp.usage.prompt_tokens,
+ resp.usage.completion_tokens,
+ )
+ content = resp.choices[0].message.content
+ if content is None:
+ raise ValueError("LLM returned None content")
+ return content
+ except Exception as exc:
+ self.logger.warning(
+ "Harness simple LLM call attempt %d/%d failed: %s",
+ attempt + 1, max_attempts, exc,
+ )
+ if attempt + 1 >= max_attempts:
+ self._log_failure("llm_call", exc, {
+ "model": model,
+ "prompt_length": len(prompt),
+ "attempts": max_attempts,
+ })
+ raise
+ time.sleep(2 ** attempt)
+ return ""
+
+ def _build_session_trace(self) -> str:
+ events: List[Dict[str, Any]] = []
+ for entry in self._action_log:
+ events.append({"type": "action", **entry})
+ for entry in self._observation_log:
+ events.append({"type": "observation", **entry})
+
+ events.sort(key=lambda e: (e.get("step", 0), 0 if e["type"] == "action" else 1))
+
+ lines: List[str] = []
+ for event in events:
+ step = event.get("step", "?")
+ if event["type"] == "action":
+ action = event.get("action", "?")
+ args = event.get("arguments", event.get("content", ""))
+ lines.append(f"[Step {step}] Action: {action}")
+ if args:
+ lines.append(f" Args: {str(args)[:500]}")
+ else:
+ action = event.get("action", "env")
+ content = event.get("content", "")
+ lines.append(f"[Step {step}] Observation from {action}:")
+ lines.append(f" {str(content)[:500]}")
+
+ return "\n".join(lines) if lines else "(No session trace recorded)"
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/harness/harness_store.py b/labs/AgentStream/exgentic/src/exgentic/agents/harness/harness_store.py
new file mode 100644
index 00000000..2c3e3a21
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/harness/harness_store.py
@@ -0,0 +1,320 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import copy
+import json
+import threading
+from dataclasses import asdict, dataclass, field
+from datetime import datetime
+from typing import Any, Dict, List, Optional
+
+
+DEFAULT_SYSTEM_PROMPT = """\
+You are an expert agent that completes tasks using available tools.
+Think step-by-step before acting.
+Use available tools to interact with the environment.
+When you are confident in your solution, use the finish/submit tool."""
+
+DEFAULT_MEMORY = ""
+
+
+@dataclass
+class HarnessSkill:
+
+ name: str
+ description: str
+ body: str
+ last_used_session: int = 0
+ created_session: int = 0
+
+
+@dataclass
+class VersionEntry:
+
+ version: int
+ session_id: str
+ session_count: int
+ ops_summary: List[str]
+ timestamp: str = ""
+ system_prompt: str = ""
+ memory: str = ""
+ skills: Dict[str, Any] = field(default_factory=dict)
+
+
+@dataclass
+class LearningEvent:
+
+ session_id: str
+ task_id: str
+ benchmark_id: str
+ ops_applied: int
+ ops_summary: List[str] = field(default_factory=list)
+ timestamp: str = ""
+
+
+class HarnessStore:
+
+ _instances: Dict[str, "HarnessStore"] = {}
+ _class_lock = threading.Lock()
+
+ def __init__(self, store_id: str) -> None:
+ self._store_id = store_id
+ self._lock = threading.Lock()
+ self.system_prompt: str = DEFAULT_SYSTEM_PROMPT
+ self.memory: str = DEFAULT_MEMORY
+ self.skills: Dict[str, HarnessSkill] = {}
+ self.skill_embeddings: Dict[str, List[float]] = {}
+ self._session_count: int = 0
+ self._history: List[LearningEvent] = []
+ self._versions: List[VersionEntry] = []
+ self._next_version: int = 0
+
+ @property
+ def store_id(self) -> str:
+ return self._store_id
+
+ @property
+ def session_count(self) -> int:
+ with self._lock:
+ return self._session_count
+
+ @property
+ def skill_count(self) -> int:
+ with self._lock:
+ return len(self.skills)
+
+ @classmethod
+ def get_or_create(
+ cls,
+ shuffle_mode: str = "isolated",
+ benchmark_id: Optional[str] = None,
+ task_group: Optional[str] = None,
+ ) -> "HarnessStore":
+ if shuffle_mode == "isolated":
+ key = f"harness_isolated_{benchmark_id or task_group or 'default'}"
+ elif shuffle_mode == "sequential":
+ key = "harness_sequential_global"
+ elif shuffle_mode == "interleaved":
+ key = "harness_interleaved_global"
+ else:
+ key = f"harness_{shuffle_mode}"
+
+ with cls._class_lock:
+ if key not in cls._instances:
+ cls._instances[key] = cls(store_id=key)
+ return cls._instances[key]
+
+ @classmethod
+ def list_stores(cls) -> Dict[str, "HarnessStore"]:
+ with cls._class_lock:
+ return dict(cls._instances)
+
+ @classmethod
+ def reset_all(cls) -> None:
+ with cls._class_lock:
+ cls._instances.clear()
+
+ def increment_session(self) -> int:
+ with self._lock:
+ self._session_count += 1
+ return self._session_count
+
+ def add_skill(self, name: str, description: str, body: str) -> None:
+ with self._lock:
+ self.skills[name] = HarnessSkill(
+ name=name,
+ description=description,
+ body=body,
+ last_used_session=self._session_count,
+ created_session=self._session_count,
+ )
+
+ def edit_skill(
+ self, name: str, description: Optional[str] = None, body: Optional[str] = None
+ ) -> bool:
+ with self._lock:
+ skill = self.skills.get(name)
+ if skill is None:
+ return False
+ if description is not None:
+ skill.description = description
+ if body is not None:
+ skill.body = body
+ return True
+
+ def delete_skill(self, name: str) -> bool:
+ with self._lock:
+ if name in self.skills:
+ del self.skills[name]
+ self.skill_embeddings.pop(name, None)
+ return True
+ return False
+
+ def touch_skills(self, names: List[str]) -> None:
+ with self._lock:
+ for name in names:
+ skill = self.skills.get(name)
+ if skill:
+ skill.last_used_session = self._session_count
+
+ def get_skill_index(self) -> List[Dict[str, str]]:
+ with self._lock:
+ return [
+ {"name": s.name, "description": s.description}
+ for s in self.skills.values()
+ ]
+
+ def list_skills(self) -> List[HarnessSkill]:
+ with self._lock:
+ return list(self.skills.values())
+
+ def edit_prompt(self, new_prompt: str) -> None:
+ with self._lock:
+ self.system_prompt = new_prompt
+
+ def edit_memory(self, new_memory: str) -> None:
+ with self._lock:
+ self.memory = new_memory
+
+ def set_embedding(self, skill_name: str, embedding: List[float]) -> None:
+ with self._lock:
+ self.skill_embeddings[skill_name] = embedding
+
+ def get_embeddings(self) -> Dict[str, List[float]]:
+ with self._lock:
+ return dict(self.skill_embeddings)
+
+ def snapshot(self) -> Dict[str, Any]:
+ with self._lock:
+ return {
+ "system_prompt": self.system_prompt,
+ "memory": self.memory,
+ "skills": copy.deepcopy(self.skills),
+ "skill_embeddings": copy.deepcopy(self.skill_embeddings),
+ }
+
+ def rollback(self, snap: Dict[str, Any]) -> None:
+ with self._lock:
+ self.system_prompt = snap["system_prompt"]
+ self.memory = snap["memory"]
+ self.skills = snap["skills"]
+ self.skill_embeddings = snap["skill_embeddings"]
+
+ def commit_version(self, session_id: str, ops_summary: List[str]) -> int:
+ with self._lock:
+ version = self._next_version
+ self._next_version += 1
+ entry = VersionEntry(
+ version=version,
+ session_id=session_id,
+ session_count=self._session_count,
+ ops_summary=ops_summary,
+ timestamp=datetime.now().isoformat(),
+ system_prompt=self.system_prompt,
+ memory=self.memory,
+ skills={name: asdict(s) for name, s in self.skills.items()},
+ )
+ self._versions.append(entry)
+ return version
+
+ def record_learning(
+ self,
+ session_id: str,
+ task_id: str,
+ benchmark_id: str,
+ ops_applied: int,
+ ops_summary: Optional[List[str]] = None,
+ ) -> None:
+ with self._lock:
+ self._history.append(LearningEvent(
+ session_id=session_id,
+ task_id=task_id,
+ benchmark_id=benchmark_id,
+ ops_applied=ops_applied,
+ ops_summary=ops_summary or [],
+ timestamp=datetime.now().isoformat(),
+ ))
+
+ def save_checkpoint(self, path: str) -> None:
+ with self._lock:
+ data = {
+ "store_id": self._store_id,
+ "session_count": self._session_count,
+ "system_prompt": self.system_prompt,
+ "memory": self.memory,
+ "skills": {name: asdict(s) for name, s in self.skills.items()},
+ "skill_embeddings": self.skill_embeddings,
+ "history": [asdict(e) for e in self._history],
+ "versions": [
+ {
+ "version": v.version,
+ "session_id": v.session_id,
+ "session_count": v.session_count,
+ "ops_summary": v.ops_summary,
+ "timestamp": v.timestamp,
+ "skill_names": list(v.skills.keys()),
+ }
+ for v in self._versions
+ ],
+ }
+ with open(path, "w", encoding="utf-8") as f:
+ json.dump(data, f, ensure_ascii=False, indent=2)
+
+ def load_checkpoint(self, path: str) -> None:
+ with open(path, "r", encoding="utf-8") as f:
+ data = json.load(f)
+ with self._lock:
+ self._session_count = data.get("session_count", 0)
+ self.system_prompt = data.get("system_prompt", DEFAULT_SYSTEM_PROMPT)
+ self.memory = data.get("memory", DEFAULT_MEMORY)
+ self.skills = {}
+ for name, sdata in data.get("skills", {}).items():
+ self.skills[name] = HarnessSkill(**{
+ k: v for k, v in sdata.items()
+ if k in HarnessSkill.__dataclass_fields__
+ })
+ self.skill_embeddings = data.get("skill_embeddings", {})
+ self._history = [
+ LearningEvent(**{
+ k: v for k, v in e.items()
+ if k in LearningEvent.__dataclass_fields__
+ })
+ for e in data.get("history", [])
+ ]
+ self._versions = []
+ for vdata in data.get("versions", []):
+ self._versions.append(VersionEntry(
+ version=vdata["version"],
+ session_id=vdata.get("session_id", ""),
+ session_count=vdata.get("session_count", 0),
+ ops_summary=vdata.get("ops_summary", []),
+ timestamp=vdata.get("timestamp", ""),
+ skills={name: {} for name in vdata.get("skill_names", [])},
+ ))
+ self._next_version = (
+ self._versions[-1].version + 1 if self._versions else 0
+ )
+
+ def save_harness_text(self, path: str) -> None:
+ with self._lock:
+ skills = list(self.skills.values())
+ lines = [
+ f"# Harness State: {self._store_id}",
+ f"## System Prompt",
+ self.system_prompt,
+ "",
+ f"## Memory",
+ self.memory,
+ "",
+ f"## Skills ({len(skills)})",
+ ]
+ for s in skills:
+ lines.append(f"### {s.name}")
+ lines.append(f"Description: {s.description}")
+ lines.append(f"Last used: session {s.last_used_session}")
+ lines.append(s.body)
+ lines.append("")
+ with open(path, "w", encoding="utf-8") as f:
+ f.write("\n".join(lines))
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/harness/prompts/__init__.py b/labs/AgentStream/exgentic/src/exgentic/agents/harness/prompts/__init__.py
new file mode 100644
index 00000000..d0286a84
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/harness/prompts/__init__.py
@@ -0,0 +1,6 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+from .inject import build_system_message
+from .evolver import EVOLVER_SYSTEM_PROMPT, build_evolution_user_message
+
+__all__ = ["build_system_message", "EVOLVER_SYSTEM_PROMPT", "build_evolution_user_message"]
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/harness/prompts/evolver.py b/labs/AgentStream/exgentic/src/exgentic/agents/harness/prompts/evolver.py
new file mode 100644
index 00000000..05a5e014
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/harness/prompts/evolver.py
@@ -0,0 +1,52 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+EVOLVER_SYSTEM_PROMPT = """\
+You are an evolution engine for an agent harness. Your job is to analyze a completed task session and improve the agent's harness (system prompt, long-term memory, and skill library) for future tasks.
+
+## Available Tools
+
+**Read tools** (use these first to inspect current state):
+- read_prompt() — read the current system prompt
+- read_memory() — read the current long-term memory document
+- list_skills() — list all skills with names and descriptions
+- read_skill(name) — read a specific skill's full body
+
+**Write tools** (use these to make changes):
+- edit_prompt(body) — replace the entire system prompt
+- edit_memory(body) — replace the entire memory document
+- add_skill(name, description, body) — add a new skill
+- edit_skill(name, description?, body?) — modify an existing skill
+- delete_skill(name) — remove a skill
+
+## Constraints
+- At most 1 edit_prompt call per session.
+- At most 1 edit_memory call per session.
+- No limit on skill operations.
+
+## Guidelines
+- First READ the current harness state, then decide what changes to make.
+- Skills should be generalizable (useful across tasks), not task-specific.
+- Memory should capture recurring patterns, proven strategies, and environment quirks.
+- System prompt changes should refine the agent's general approach.
+- Do NOT duplicate information already present in the harness.
+- If no changes are needed, simply stop without calling any write tools.
+"""
+
+
+def build_evolution_user_message(
+ task: str,
+ injected_skill_names: list[str],
+ trajectory: str,
+) -> str:
+ parts = [
+ "## This Session\n",
+ f"### Task\n{task}\n",
+ f"### Skills Injected\n{', '.join(injected_skill_names) if injected_skill_names else '(None)'}\n",
+ f"### Session Trajectory\n{trajectory}\n",
+ "\n---\n",
+ "Analyze the session above. Read the current harness state using the read tools, "
+ "then decide what changes (if any) would improve the agent's future performance. "
+ "Make changes using the write tools, or stop if no changes are needed.",
+ ]
+ return "\n".join(parts)
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/harness/prompts/inject.py b/labs/AgentStream/exgentic/src/exgentic/agents/harness/prompts/inject.py
new file mode 100644
index 00000000..9b1e237b
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/harness/prompts/inject.py
@@ -0,0 +1,30 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+from typing import List, Tuple
+
+from ..harness_store import HarnessSkill
+
+
+def build_system_message(
+ system_prompt: str,
+ memory: str,
+ retrieved_skills: List[Tuple[HarnessSkill, float]],
+) -> str:
+ parts = [system_prompt]
+
+ if memory and memory.strip():
+ parts.append("\n\n## Long-Term Memory\n")
+ parts.append(memory)
+
+ if retrieved_skills:
+ parts.append("\n\n## Retrieved Skills\n")
+ for skill, _score in retrieved_skills:
+ parts.append(f"### Skill: {skill.name}")
+ parts.append(f"*{skill.description}*\n")
+ parts.append(skill.body)
+ parts.append("")
+
+ return "\n".join(parts)
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/harness/retriever.py b/labs/AgentStream/exgentic/src/exgentic/agents/harness/retriever.py
new file mode 100644
index 00000000..2299a2cd
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/harness/retriever.py
@@ -0,0 +1,93 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import logging
+import math
+from typing import List, Tuple
+
+from .harness_store import HarnessSkill, HarnessStore
+
+logger = logging.getLogger(__name__)
+
+_st_model = None
+_st_model_name = None
+
+
+def _get_st_model(model_name: str = "all-MiniLM-L6-v2"):
+ global _st_model, _st_model_name
+ if _st_model is None or _st_model_name != model_name:
+ from sentence_transformers import SentenceTransformer
+ logger.info("Loading SentenceTransformer model: %s", model_name)
+ _st_model = SentenceTransformer(model_name)
+ _st_model_name = model_name
+ return _st_model
+
+
+def compute_embedding(
+ text: str, model: str = "all-MiniLM-L6-v2"
+) -> List[float]:
+ try:
+ st = _get_st_model(model)
+ vec = st.encode([text])[0]
+ return vec.tolist()
+ except Exception as exc:
+ logger.error("Local embedding failed: %s", exc)
+ return []
+
+
+def cosine_similarity(a: List[float], b: List[float]) -> float:
+ if not a or not b or len(a) != len(b):
+ return 0.0
+ dot = sum(x * y for x, y in zip(a, b))
+ norm_a = math.sqrt(sum(x * x for x in a))
+ norm_b = math.sqrt(sum(x * x for x in b))
+ if norm_a == 0 or norm_b == 0:
+ return 0.0
+ return dot / (norm_a * norm_b)
+
+
+def retrieve_skills(
+ task_text: str,
+ store: HarnessStore,
+ top_k: int = 5,
+ embedding_model: str = "all-MiniLM-L6-v2",
+) -> List[Tuple[HarnessSkill, float]]:
+ if store.skill_count == 0:
+ return []
+
+ query_embedding = compute_embedding(task_text, model=embedding_model)
+ if not query_embedding:
+ logger.warning("Failed to compute query embedding, returning no skills")
+ return []
+
+ skills = store.list_skills()
+ embeddings = store.get_embeddings()
+ for skill in skills:
+ if skill.name not in embeddings:
+ emb = compute_embedding(skill.description, model=embedding_model)
+ if emb:
+ store.set_embedding(skill.name, emb)
+
+ embeddings = store.get_embeddings()
+ scored: List[Tuple[HarnessSkill, float]] = []
+ for skill in skills:
+ emb = embeddings.get(skill.name)
+ if emb:
+ score = cosine_similarity(query_embedding, emb)
+ scored.append((skill, score))
+
+ scored.sort(key=lambda x: x[1], reverse=True)
+ results = scored[:top_k]
+
+ if results:
+ logger.info(
+ "Skill retrieval: query='%s...' → retrieved %d/%d skills: %s",
+ task_text[:60], len(results), len(skills),
+ [(s.name, f"{score:.3f}") for s, score in results],
+ )
+ else:
+ logger.info("Skill retrieval: no skills scored above 0")
+
+ return results
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/litellm_tool_calling/__init__.py b/labs/AgentStream/exgentic/src/exgentic/agents/litellm_tool_calling/__init__.py
new file mode 100644
index 00000000..367fec41
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/litellm_tool_calling/__init__.py
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/litellm_tool_calling/instance.py b/labs/AgentStream/exgentic/src/exgentic/agents/litellm_tool_calling/instance.py
new file mode 100644
index 00000000..9d44cbb1
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/litellm_tool_calling/instance.py
@@ -0,0 +1,486 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from __future__ import annotations
+
+import json
+import time
+from typing import Any, Union
+
+import litellm
+litellm.cache = None # Disable LiteLLM
+from litellm import (
+ ChatCompletionAssistantMessage,
+ ChatCompletionDeveloperMessage,
+ ChatCompletionToolMessage,
+ ChatCompletionUserMessage,
+)
+
+from ...core.agent_instance import AgentInstance
+from ...core.context import get_context
+from ...core.types import (
+ Action,
+ ActionType,
+ Message,
+ MessageAction,
+ MessageObservation,
+ MessagePayload,
+ ModelSettings,
+ Observation,
+ RetryStrategy,
+)
+from ...integrations.litellm.health import check_model_accessible_sync
+from ...utils.cost import LiteLLMCostReport
+from ...utils.settings import get_settings
+from .utils import ToolCall, ToolsActionsRegistry
+
+settings = get_settings()
+
+
+class NonRetryableCompletionError(ValueError):
+ """Raised when a completion response should not be retried."""
+
+
+class LiteLLMToolCallingAgentInstance(AgentInstance):
+ """Ultra-simple tool-calling agent.
+
+ - If the model produces tool_calls, convert them directly to Actions without schema verification.
+ - If the model produces a plain assistant message, interpret it as a `message` action
+ (even if that tool is not advertised) and emit a corresponding Action.
+ """
+
+ def __init__(
+ self,
+ session_id: str,
+ model: str = "gpt-4o-mini",
+ max_steps: int = 150,
+ enable_tool_shortlisting: bool = True,
+ max_selected_tools: int = 30,
+ model_settings: ModelSettings | None = None,
+ allow_truncated_messages: bool = False,
+ ):
+ super().__init__(session_id)
+ self.model = model
+ self.max_steps = max_steps
+ self.enable_tool_shortlisting = enable_tool_shortlisting
+ self.max_selected_tools = max_selected_tools
+ if model_settings is None:
+ self.model_settings = ModelSettings()
+ elif isinstance(model_settings, ModelSettings):
+ self.model_settings = model_settings
+ else:
+ raise ValueError("model_settings must be a ModelSettings instance.")
+ self._allow_truncated_messages = allow_truncated_messages
+ self._use_cache = settings.litellm_caching
+ self.logger.debug(
+ "LiteLLM cache %s (dir=%s)",
+ "enabled" if self._use_cache else "disabled",
+ settings.resolved_litellm_cache_dir(),
+ )
+
+ self.messages: list[
+ Union[
+ ChatCompletionAssistantMessage,
+ ChatCompletionToolMessage,
+ ChatCompletionUserMessage,
+ ]
+ ] = []
+ self._step_count = 0
+ self._cost_data = LiteLLMCostReport.initialize_empty(model_name=self.model)
+
+ # Check model accessibility
+ check_model_accessible_sync(self.model, logger=self.logger)
+
+ def start(self, task, context, actions):
+ """Receive work payload, build tool registry, and seed conversation."""
+ super().start(task, context, actions)
+
+ for a in self.actions:
+ if not isinstance(a, ActionType):
+ raise ValueError("Invalid action type provided to agent")
+
+ self._all_actions: list[ActionType] = list(self.actions)
+ self._registry = ToolsActionsRegistry(self._all_actions)
+
+ # Seed conversation with task + context
+ content_parts: list[Any] = []
+ ctx = ""
+ if self.context:
+ for k, v in self.context.items():
+ if isinstance(v, dict) and v.get("type") == "image_url":
+ content_parts.append({"type": "image_url", "image_url": {"url": v["data"], "detail": "high"}})
+ else:
+ ctx += f"\n<{k}>\n{v}\n"
+
+ text_content = f"{self.task}\n{ctx}"
+ if content_parts:
+ content_parts.insert(0, {"type": "text", "text": text_content})
+ self._add_message(ChatCompletionUserMessage(role="user", content=content_parts))
+ else:
+ self._add_message(ChatCompletionUserMessage(role="user", content=text_content))
+
+ def _register_cost(self, usage: litellm.Usage):
+ self._cost_data.update_cost_from_tokens(usage.prompt_tokens, usage.completion_tokens)
+
+ def _add_message(self, message):
+ self.logger.info(f"Adding message to chat history: {message}")
+ self.messages.append(message)
+
+ def _observe(self, observation: Observation | None):
+ if observation is None:
+ self.logger.info("Skipping observation: None")
+ return
+
+ observations = observation.to_observation_list()
+ if observation.is_empty():
+ # Preserve tool results even when the result payload is empty.
+ if not any(obs.invoking_actions for obs in observations):
+ self.logger.info("Skipping observation: empty with no invoking_actions")
+ return
+
+ for obs in observations:
+ # Structured user messages: add and move on
+ if isinstance(obs, MessageObservation) and isinstance(obs.result, MessagePayload):
+ self._add_message(ChatCompletionUserMessage(role="user", content=obs.result.message))
+ continue
+
+ if len(obs.invoking_actions) > 0:
+ invoking = obs.invoking_actions[0]
+ if invoking.name == "message":
+ # Fallback: treat as user-visible content
+ self._add_message(ChatCompletionUserMessage(role="user", content=str(obs)))
+ continue
+ action_id = invoking.id
+ tool_call_id = invoking.id
+ if not (isinstance(tool_call_id, str) and tool_call_id.startswith("call_")):
+ tool_call_id = self._registry.action_id_to_tool_call_id.get(action_id)
+ if tool_call_id is None:
+ raise RuntimeError(f"Unable to map tool call id for action {action_id}")
+ value = obs.result
+
+ # Extract image_url entries for vision support
+ image_parts: list[dict] = []
+ if isinstance(value, dict):
+ text_value = {}
+ for k, v in value.items():
+ if isinstance(v, dict) and v.get("type") == "image_url":
+ image_parts.append({"type": "image_url", "image_url": {"url": v["data"], "detail": "high"}})
+ else:
+ text_value[k] = v
+ if image_parts:
+ value = text_value # Tool result without image data
+
+ try:
+ content = json.dumps(value, ensure_ascii=False, separators=(",", ":"))
+ except TypeError:
+ content = str(value)
+ self._add_message(ChatCompletionToolMessage(role="tool", tool_call_id=tool_call_id, content=content))
+
+ # Add images as follow-up user message for vision models
+ if image_parts:
+ image_parts.insert(0, {"type": "text", "text": "Observation screenshot:"})
+ self._add_message(ChatCompletionUserMessage(role="user", content=image_parts))
+ else:
+ # Initial observation (no invoking actions) — handle vision content
+ value = obs.result
+ image_parts: list[dict] = []
+ if isinstance(value, dict):
+ text_value = {}
+ for k, v in value.items():
+ if isinstance(v, dict) and v.get("type") == "image_url":
+ image_parts.append({"type": "image_url", "image_url": {"url": v["data"], "detail": "high"}})
+ else:
+ text_value[k] = v
+ if image_parts:
+ value = text_value
+ if image_parts:
+ try:
+ text_content = json.dumps(value, ensure_ascii=False, separators=(",", ":"))
+ except TypeError:
+ text_content = str(value)
+ content_parts: list[dict] = [{"type": "text", "text": f"Initial observation: {text_content}"}]
+ content_parts.extend(image_parts)
+ self._add_message(ChatCompletionUserMessage(role="user", content=content_parts))
+ else:
+ self._add_message(ChatCompletionUserMessage(role="user", content=str(obs)))
+
+ def _assistant_tools(self) -> list[dict[str, Any]]:
+ """Returns list of available tools in openai format.
+
+ If the number of available tools is less the max_selected_tools parameter,
+ returns all available tools.
+
+ Otherwise, calls an LLM to shortlist the tools, and find the most relevant one for
+ the current stage in the chat,
+
+ """
+ tools = self._registry.openai_tools()
+
+ if not self.enable_tool_shortlisting:
+ self.logger.info(
+ "Tool shortlisting disabled: returning all %d tools",
+ len(tools),
+ )
+ return tools
+
+ if len(tools) <= self.max_selected_tools:
+ self.logger.info(
+ "Tool shortlist bypassed: %d tools <= max_selected_tools",
+ len(tools),
+ )
+ return tools
+ self.logger.info(
+ "Selecting tools: %d available -> top %d",
+ len(tools),
+ self.max_selected_tools,
+ )
+
+ names = [tool["function"]["name"] for tool in tools]
+
+ names_str = ""
+ for tool in tools:
+ names_str += f"\n- {tool['function']['name']}: {tool['function']['description']}"
+
+ history_text = self._render_history_for_shortlist()
+ self.logger.info("Tool shortlist history chars: %d", len(history_text))
+
+ dev = ChatCompletionDeveloperMessage(
+ role="developer",
+ content=(
+ f"Please before providing your next move list the names of the top "
+ f"{self.max_selected_tools} tools that are somewhat relevant for the next step, "
+ "ordered by relevancy (most to least). Return ONLY a JSON object with this shape: "
+ '{\n "tools": ["tool_name_1", "tool_name_2", ...]\n}.\n'
+ f"Choose from these tools only: {names_str}.\n"
+ f"Do not call any of those tools just return the list of the top "
+ f"{self.max_selected_tools} relevant tools names in the required format."
+ ),
+ )
+ history = ChatCompletionUserMessage(
+ role="user",
+ content=f"Conversation so far (plain text):\n{history_text}",
+ )
+
+ try:
+ response = self._completion(
+ model=self.model,
+ messages=[dev, history],
+ caching=self._use_cache,
+ )
+ except Exception as exc:
+ self.logger.warning("Tool shortlisting LLM call failed: %s", exc)
+ return tools[: self.max_selected_tools]
+
+ self._register_cost(response.usage)
+
+ text = response.choices[0].message["content"]
+
+ if text is None:
+ text = str(response.choices[0].message)
+
+ self.logger.info("Tool shortlist model response: %s", text)
+
+ positions = []
+ for name in names:
+ idx = text.find(name)
+ if idx != -1:
+ positions.append((idx, name))
+
+ if len(positions) == 0:
+ selected_tools = tools[: self.max_selected_tools]
+ self.logger.info(
+ "Tool shortlist fallback: %d -> %d (no matches in model response)",
+ len(tools),
+ len(selected_tools),
+ )
+ if len(selected_tools) == 0:
+ self.logger.warning("Tool shortlist reduced to 0 tools")
+ return selected_tools
+
+ # Sort tools by the order they appear in the model response
+ positions.sort(key=lambda x: x[0])
+
+ ordered_tools = [name for _, name in positions]
+
+ selected_names = ordered_tools[: self.max_selected_tools]
+ name_to_tool = {tool["function"]["name"]: tool for tool in tools}
+ selected_tools = [name_to_tool[name] for name in selected_names]
+ self.logger.info(
+ "Tool shortlist from model: %d -> %d",
+ len(tools),
+ len(selected_tools),
+ )
+ if len(selected_tools) == 0:
+ self.logger.warning("Tool shortlist reduced to 0 tools")
+ return selected_tools
+
+ def _render_history_for_shortlist(self) -> str:
+ parts = []
+ for message in self.messages:
+ msg = self._message_to_dict(message)
+ role = msg.get("role") or "unknown"
+ if role == "tool":
+ content = msg.get("content", "")
+ parts.append(f"tool: {content}")
+ continue
+ content = msg.get("content")
+ if content:
+ parts.append(f"{role}: {content}")
+ tool_calls = msg.get("tool_calls") or []
+ for tool_call in tool_calls:
+ function = tool_call.get("function") or {}
+ name = function.get("name") or tool_call.get("name")
+ arguments = function.get("arguments")
+ parts.append(f"{role} tool_call: {name}({arguments})")
+ return "\n".join(parts)
+
+ @staticmethod
+ def _message_to_dict(message: Any) -> dict[str, Any]:
+ if isinstance(message, dict):
+ return message
+ if hasattr(message, "model_dump"):
+ return message.model_dump()
+ if hasattr(message, "dict"):
+ return message.dict()
+ raise TypeError(f"Unsupported message type: {type(message).__name__}")
+
+ def _extract_tool_calls(self, message: litellm.Message) -> list[ToolCall]:
+ """Extract tool calls from the message object returned from the litellm call."""
+ if not message.tool_calls:
+ return []
+
+ tool_calls: list[ToolCall] = []
+
+ for tool_call in message.tool_calls:
+ tool_calls.append(
+ {
+ "name": tool_call.function.name,
+ "arguments": tool_call.function.arguments,
+ "id": tool_call.id,
+ }
+ )
+
+ return tool_calls
+
+ def react(self, observation: Observation | None) -> Action | None:
+ self._step_count += 1
+ if self._step_count > self.max_steps:
+ self.logger.warning("Finished: max steps reached (%d)", self.max_steps)
+ return None
+
+ self._observe(observation)
+
+ response = self._completion(
+ model=self.model,
+ messages=self.messages,
+ tools=self._assistant_tools(),
+ caching=self._use_cache,
+ )
+
+ self._register_cost(response.usage)
+
+ choice = response["choices"][0]
+ message = choice["message"]
+ finish_reason = choice.get("finish_reason")
+
+ if finish_reason == "tool_calls":
+ tool_calls = self._extract_tool_calls(message)
+ self._add_message(
+ ChatCompletionAssistantMessage(
+ role="assistant",
+ tool_calls=[
+ {
+ "id": tool_call["id"],
+ "type": "function",
+ "function": {
+ "name": tool_call["name"],
+ "arguments": tool_call["arguments"],
+ },
+ }
+ for tool_call in tool_calls
+ ],
+ )
+ )
+ actions = self._registry.tool_calls_to_action(tool_calls)
+ else:
+ actions = MessageAction(arguments=Message(content=message.content))
+ self._add_message(
+ ChatCompletionAssistantMessage(
+ role="assistant",
+ content=message.content,
+ )
+ )
+
+ self.logger.info(f"Invoking action: {actions}")
+ return actions
+
+ def _completion(self, **kwargs):
+ call_kwargs = self.model_settings.model_dump(
+ exclude_none=True,
+ exclude={"num_retries", "retry_after", "retry_strategy"},
+ )
+ call_kwargs.update(kwargs)
+ # Use 'metadata' parameter instead of 'litellm_metadata' - LiteLLM passes this to callbacks
+ call_kwargs.setdefault("metadata", {})["context"] = get_context()
+ return self._completion_with_retries(call_kwargs)
+
+ def _completion_with_retries(self, call_kwargs: dict[str, Any]):
+ num_retries = self.model_settings.num_retries or 0
+ max_attempts = max(1, num_retries + 1)
+ for attempt in range(max_attempts):
+ try:
+ response = litellm.completion(max_retries=0, **call_kwargs)
+ self._raise_if_invalid_completion(response)
+ return response
+ except NonRetryableCompletionError:
+ raise
+ except Exception as exc:
+ if attempt >= num_retries:
+ raise
+ delay = self.model_settings.retry_after
+ retry_strategy = self.model_settings.retry_strategy.value
+ if retry_strategy == RetryStrategy.EXPONENTIAL_BACKOFF.value:
+ delay *= 2**attempt
+ self.logger.warning(
+ "LiteLLM completion failed (attempt %d/%d): %s",
+ attempt + 1,
+ num_retries + 1,
+ exc,
+ )
+ if delay > 0:
+ time.sleep(delay)
+ return None
+
+ def _raise_if_invalid_completion(self, response: Any) -> None:
+ try:
+ choice = response["choices"][0]
+ message = choice["message"]
+ finish_reason = choice.get("finish_reason")
+ except Exception:
+ return
+
+ if finish_reason == "length" and not self._allow_truncated_messages:
+ self.logger.error(
+ "LiteLLM completion truncated (finish_reason=length). Raw response: %s",
+ response,
+ )
+ raise NonRetryableCompletionError(
+ "LiteLLM completion truncated (finish_reason=length). "
+ "To allow truncated responses, configure the agent with "
+ "allow_truncated_messages=True, or increase max_tokens."
+ )
+
+ if finish_reason != "tool_calls":
+ if message is None or message.content is None:
+ self.logger.error(
+ "LiteLLM completion missing assistant content " "(finish_reason=%s). Raw response: %s",
+ finish_reason,
+ response,
+ )
+ raise ValueError("LiteLLM completion missing assistant content.")
+
+ def close(self) -> None:
+ pass
+
+ def get_cost(self) -> LiteLLMCostReport:
+ return self._cost_data
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/litellm_tool_calling/litellm_tool_calling_agent.py b/labs/AgentStream/exgentic/src/exgentic/agents/litellm_tool_calling/litellm_tool_calling_agent.py
new file mode 100644
index 00000000..c6172608
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/litellm_tool_calling/litellm_tool_calling_agent.py
@@ -0,0 +1,54 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from __future__ import annotations
+
+from typing import Any, ClassVar
+
+from ...core.agent import Agent
+from ...core.types import ModelSettings
+
+
+class LiteLLMToolCallingAgent(Agent):
+ """Agent factory that always assigns the message-to-user mapping variant."""
+
+ display_name: ClassVar[str] = "LiteLLM Tool Calling"
+ slug_name: ClassVar[str] = "tool_calling"
+
+ model: str = "watsonx/openai/gpt-oss-120b"
+ max_steps: int = 150
+ enable_tool_shortlisting: bool = False
+ max_selected_tools: int = 30
+ model_settings: ModelSettings | None = None
+ allow_truncated_messages: bool = False
+
+ @classmethod
+ def _get_instance_class(cls):
+ from .instance import LiteLLMToolCallingAgentInstance
+
+ return LiteLLMToolCallingAgentInstance
+
+ @classmethod
+ def _get_instance_class_ref(cls) -> str:
+ return "exgentic.agents.litellm_tool_calling.instance:LiteLLMToolCallingAgentInstance"
+
+ @property
+ def model_name(self) -> str: # type: ignore[override]
+ return str(self.model).split("/")[-1]
+
+ def get_models_names(self) -> list[str]: # type: ignore[override]
+ return [str(self.model)]
+
+ def _get_instance_kwargs(
+ self,
+ session_id: str,
+ ) -> dict[str, Any]:
+ return {
+ "session_id": session_id,
+ "model": self.model,
+ "enable_tool_shortlisting": self.enable_tool_shortlisting,
+ "max_selected_tools": self.max_selected_tools,
+ "max_steps": self.max_steps,
+ "model_settings": self.model_settings,
+ "allow_truncated_messages": self.allow_truncated_messages,
+ }
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/litellm_tool_calling/utils.py b/labs/AgentStream/exgentic/src/exgentic/agents/litellm_tool_calling/utils.py
new file mode 100644
index 00000000..03aac91a
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/litellm_tool_calling/utils.py
@@ -0,0 +1,135 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from __future__ import annotations
+
+import json
+from typing import Any, Optional
+
+from typing_extensions import TypedDict
+
+from ...core.actions import build_action, build_unknown_action
+from ...core.types import (
+ Action,
+ ActionType,
+ ParallelAction,
+ SingleAction,
+ SingleObservation,
+)
+
+
+class PartialAction(SingleAction):
+ arguments: dict
+
+
+class ToolCall(TypedDict):
+ name: str
+ arguments: str
+ id: str
+
+
+def extract_arguments(action_type: ActionType):
+ return action_type.arguments
+
+
+class ToolsActionsRegistry:
+ _MAX_SAFE_SCHEMA_INT = 2_147_483_647
+
+ @classmethod
+ def _clamp_schema_ints(cls, obj):
+ if isinstance(obj, dict):
+ return {k: cls._clamp_schema_ints(v) for k, v in obj.items()}
+ if isinstance(obj, list):
+ return [cls._clamp_schema_ints(v) for v in obj]
+ if isinstance(obj, int) and not isinstance(obj, bool):
+ if obj > cls._MAX_SAFE_SCHEMA_INT:
+ return cls._MAX_SAFE_SCHEMA_INT
+ if isinstance(obj, float):
+ if obj > cls._MAX_SAFE_SCHEMA_INT:
+ return float(cls._MAX_SAFE_SCHEMA_INT)
+ return obj
+
+ @staticmethod
+ def format_observation(observation: SingleObservation) -> str:
+ """Serialize observation.result to JSON if possible to satisfy tool message requirements."""
+ value = observation.result
+ try:
+ return json.dumps(value, ensure_ascii=False, separators=(",", ":"))
+ except TypeError:
+ return str(value)
+
+ def __init__(self, actions: list[ActionType]):
+ self.action_types: list[ActionType] = []
+ self.name_to_action: dict[str, ActionType] = {}
+ self.action_id_to_tool_call_id: dict[str, str] = {}
+ for action in actions:
+ self.add_action(action)
+
+ def add_action(self, action: ActionType):
+ if not isinstance(action, ActionType):
+ raise ValueError("bad action")
+ self.action_types.append(action)
+ self.name_to_action[action.name] = action
+
+ def openai_tools(self) -> list[dict[str, Any]]:
+ tools: list[dict[str, Any]] = []
+ for action in self.action_types:
+ # Skip non-environment messaging actions; agents handle messaging flow
+ if action.is_message:
+ continue
+ arguments_type = extract_arguments(action)
+ schema = arguments_type.model_json_schema() # type: ignore[attr-defined]
+ # Bedrock rejects oversized integer values in tool schemas.
+ schema = self._clamp_schema_ints(schema)
+ tools.append(
+ {
+ "type": "function",
+ "function": {
+ "name": action.name,
+ "description": action.description,
+ "parameters": schema,
+ },
+ }
+ )
+ tools.sort(
+ key=lambda tool: (
+ tool.get("type", ""),
+ tool.get("function", {}).get("name", ""),
+ )
+ )
+ return tools
+
+ def _tool_call_to_single_action(self, tool_call: ToolCall) -> SingleAction:
+ name = tool_call["name"]
+ action_type = self.name_to_action.get(name)
+
+ action_id = tool_call.get("id")
+ if action_type:
+ action = build_action(action_type, tool_call["arguments"], action_id=action_id)
+ else:
+ action = build_unknown_action(name, tool_call.get("arguments", {}), action_id=action_id)
+
+ if "id" not in tool_call:
+ tool_call["id"] = action.id
+
+ self.action_id_to_tool_call_id[action.id] = tool_call["id"]
+
+ return action
+
+ def tool_calls_to_action(self, tool_calls: list[ToolCall]) -> Optional[Action]:
+ actions: list[SingleAction] = []
+ for tool_call in tool_calls:
+ actions.append(self._tool_call_to_single_action(tool_call))
+ if len(actions) == 0:
+ return None
+ if len(actions) == 1:
+ return actions[0]
+ return ParallelAction(actions=actions)
+
+
+def tool_call_to_dict(tool_call):
+ return {
+ "function": vars(tool_call.function),
+ "id": tool_call.id,
+ "type": tool_call.type,
+ }
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/openai/__init__.py b/labs/AgentStream/exgentic/src/exgentic/agents/openai/__init__.py
new file mode 100644
index 00000000..367fec41
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/openai/__init__.py
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/openai/instance.py b/labs/AgentStream/exgentic/src/exgentic/agents/openai/instance.py
new file mode 100644
index 00000000..973f23ee
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/openai/instance.py
@@ -0,0 +1,307 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+import asyncio
+import logging
+from typing import Any
+
+import httpx
+from agents import Agent as OpenAIAgent
+from agents import Runner
+from agents.extensions.models.litellm_model import LitellmModel
+from agents.lifecycle import RunHooksBase
+from agents.mcp import MCPServerStreamableHttp, MCPServerStreamableHttpParams
+from agents.model_settings import ModelSettings as OpenAIModelSettings
+from agents.model_settings import Reasoning
+from agents.run import RunConfig
+from agents.usage import Usage
+
+from ...adapters.agents.mcp_agent import MCPAgentInstance
+from ...core.context import get_context
+from ...core.types import ModelSettings, RetryStrategy
+from ...integrations.litellm.health import acheck_model_accessible
+from ...observers.logging import (
+ attach_library_logger_to_handler,
+ restore_library_logger,
+)
+from ...utils.cost import UpdatableCostReport, litellm_tokens_cost
+from ...utils.settings import get_settings
+from ...utils.sync import run_sync
+from .openai_mcp_agent import MCPConfig
+
+settings = get_settings()
+
+
+class _UsageRunHooks(RunHooksBase[dict[str, Any], OpenAIAgent]):
+ def __init__(self, record_usage) -> None:
+ super().__init__()
+ self._record_usage = record_usage
+
+ async def on_llm_end(self, context, agent, response) -> None:
+ self._record_usage(response.usage)
+
+
+class RetryingLitellmModel(LitellmModel):
+ def __init__(
+ self,
+ model: str,
+ *,
+ num_retries: int,
+ retry_after: float,
+ retry_strategy: str | RetryStrategy,
+ ):
+ super().__init__(model=model)
+ if isinstance(retry_strategy, RetryStrategy):
+ retry_strategy = retry_strategy.value
+ if retry_strategy not in ("exponential_backoff_retry", "constant_retry"):
+ raise ValueError(f"Unsupported retry_strategy: {retry_strategy}")
+ if num_retries < 0:
+ raise ValueError("num_retries must be >= 0")
+ if retry_after < 0:
+ raise ValueError("retry_after must be >= 0")
+ self._num_retries = num_retries
+ self._retry_after = retry_after
+ self._retry_strategy = retry_strategy
+
+ async def _fetch_response(self, *args, **kwargs):
+ # Inject context for OTEL tracing via model_settings.metadata
+ # The parent class extracts metadata from model_settings and passes it to litellm.acompletion()
+ # Note: metadata must be Dict[str, str], so we serialize Context fields individually
+ ctx = get_context()
+
+ # model_settings is the 3rd positional argument (index 2)
+ if len(args) > 2:
+ model_settings = args[2]
+ if model_settings.metadata is None:
+ model_settings.metadata = {}
+
+ # Serialize Context fields as individual string metadata entries
+ model_settings.metadata["exgentic_ctx_run_id"] = ctx.run_id
+ model_settings.metadata["exgentic_ctx_output_dir"] = ctx.output_dir
+ model_settings.metadata["exgentic_ctx_cache_dir"] = ctx.cache_dir
+ if ctx.session_id is not None:
+ model_settings.metadata["exgentic_ctx_session_id"] = ctx.session_id
+ if ctx.task_id is not None:
+ model_settings.metadata["exgentic_ctx_task_id"] = ctx.task_id
+ model_settings.metadata["exgentic_ctx_role"] = ctx.role.value
+ if ctx.otel_context is not None:
+ model_settings.metadata["exgentic_ctx_otel_trace_id"] = ctx.otel_context.trace_id
+ model_settings.metadata["exgentic_ctx_otel_span_id"] = ctx.otel_context.span_id
+
+ for attempt in range(self._num_retries + 1):
+ try:
+ return await super()._fetch_response(*args, **kwargs)
+ except Exception as exc:
+ if attempt >= self._num_retries:
+ raise
+ delay = self._retry_after
+ if self._retry_strategy == RetryStrategy.EXPONENTIAL_BACKOFF.value:
+ delay *= 2**attempt
+ logging.getLogger(__name__).warning(
+ "OpenAI MCP LiteLLM call failed (attempt %d/%d): %s",
+ attempt + 1,
+ self._num_retries + 1,
+ exc,
+ )
+ if delay > 0:
+ await asyncio.sleep(delay)
+ return None
+
+
+class OpenAIMCPAgentInstance(MCPAgentInstance):
+ """OpenAI Agents SDK + MCP (sync entrypoint, async core)."""
+
+ def __init__(
+ self,
+ session_id: str,
+ model_id: str,
+ max_steps: int = 150,
+ model_settings: ModelSettings | None = None,
+ mcp_config: MCPConfig | dict | None = None,
+ ):
+ super().__init__(session_id)
+ self.model_id = model_id
+ self.max_steps = max_steps
+ if model_settings is None:
+ self.model_settings = ModelSettings()
+ elif isinstance(model_settings, ModelSettings):
+ self.model_settings = model_settings
+ else:
+ raise ValueError("model_settings must be a ModelSettings instance.")
+ if mcp_config is None:
+ self.mcp_config = MCPConfig()
+ elif isinstance(mcp_config, dict):
+ self.mcp_config = MCPConfig(**mcp_config)
+ else:
+ self.mcp_config = mcp_config
+ self._total_input_tokens = 0
+ self._total_output_tokens = 0
+ self._model_access_checked = False
+
+ async def _check_model_access_once(self) -> None:
+ if self._model_access_checked or self.mcp_config.skip_health_check:
+ return
+ self.logger.info("Running LiteLLM model health check (model=%s)", self.model_id)
+ await acheck_model_accessible(self.model_id)
+ self._model_access_checked = True
+
+ def _record_usage(self, usage: Usage | None) -> None:
+ if usage is None:
+ return
+ self._total_input_tokens += usage.input_tokens
+ self._total_output_tokens += usage.output_tokens
+
+ def run_mcp_agent(self, mcp_host: str, mcp_port: int) -> Any:
+ # Run async core on the shared loop (sync API)
+ return run_sync(self.run_mcp_agent_async(mcp_host, mcp_port), timeout=600.0)
+
+ async def run_mcp_agent_async(self, mcp_host: str, mcp_port: int) -> Any:
+ RunConfig.tracing_disabled = True
+
+ prompt = self._build_prompt()
+
+ file_handler = next(
+ (h for h in self.logger.handlers if isinstance(h, logging.FileHandler)),
+ None,
+ )
+ logger_states: list[tuple] = []
+
+ try:
+ if file_handler:
+ logger_states += [
+ attach_library_logger_to_handler("agents", file_handler),
+ attach_library_logger_to_handler(__name__, file_handler),
+ ]
+ logging.getLogger("agents").setLevel(logging.INFO)
+ await self._check_model_access_once()
+
+ # Create custom httpx client factory with extended timeout
+ def httpx_client_factory(headers=None, timeout=None, auth=None):
+ if (
+ self.mcp_config.http_timeout_seconds is None
+ and self.mcp_config.sse_read_timeout_seconds is None
+ and self.mcp_config.http_connect_timeout_seconds is None
+ ):
+ client_timeout = httpx.Timeout(None)
+ else:
+ client_timeout = httpx.Timeout(
+ self.mcp_config.http_timeout_seconds,
+ connect=self.mcp_config.http_connect_timeout_seconds,
+ read=self.mcp_config.sse_read_timeout_seconds,
+ )
+ return httpx.AsyncClient(
+ headers=headers,
+ timeout=client_timeout,
+ auth=auth,
+ )
+
+ mcp_params: dict[str, Any] = {
+ "url": f"http://{mcp_host}:{mcp_port}/mcp",
+ "httpx_client_factory": httpx_client_factory,
+ "terminate_on_close": self.mcp_config.terminate_on_close,
+ }
+ if self.mcp_config.headers is not None:
+ mcp_params["headers"] = self.mcp_config.headers
+ if self.mcp_config.http_timeout_seconds is not None:
+ mcp_params["timeout"] = self.mcp_config.http_timeout_seconds
+ if self.mcp_config.sse_read_timeout_seconds is not None:
+ mcp_params["sse_read_timeout"] = self.mcp_config.sse_read_timeout_seconds
+
+ async with MCPServerStreamableHttp(
+ params=MCPServerStreamableHttpParams(**mcp_params),
+ cache_tools_list=self.mcp_config.cache_tools_list,
+ name=self.mcp_config.name,
+ client_session_timeout_seconds=self.mcp_config.client_session_timeout_seconds,
+ use_structured_content=self.mcp_config.use_structured_content,
+ max_retry_attempts=self.mcp_config.max_retry_attempts,
+ retry_backoff_seconds_base=self.mcp_config.retry_backoff_seconds_base,
+ message_handler=self.mcp_config.message_handler,
+ ) as mcp_server:
+ temperature = self.model_settings.temperature
+ reasoning_effort = self.model_settings.reasoning_effort
+ openai_model_settings = OpenAIModelSettings(
+ temperature=temperature if temperature is not None else 1.0,
+ max_tokens=self.model_settings.max_tokens,
+ top_p=self.model_settings.top_p,
+ reasoning=(Reasoning(effort=reasoning_effort) if reasoning_effort is not None else None),
+ )
+ num_retries = self.model_settings.num_retries or 0
+ retry_after = self.model_settings.retry_after
+ retry_strategy = self.model_settings.retry_strategy.value
+ openai_model_settings.extra_args = {
+ "caching": settings.litellm_caching,
+ "max_retries": 0 if num_retries > 0 else 5,
+ }
+ agent = OpenAIAgent(
+ name="Assistant",
+ instructions=prompt,
+ model=RetryingLitellmModel(
+ model=self.model_id,
+ num_retries=num_retries,
+ retry_after=retry_after,
+ retry_strategy=retry_strategy,
+ ),
+ model_settings=openai_model_settings,
+ mcp_servers=[mcp_server],
+ )
+ self.logger.info(
+ "Starting OpenAI MCP agent run (model=%s, task=%s, max_turns=%s)",
+ self.model_id,
+ self.task,
+ self.max_steps,
+ )
+ hooks = _UsageRunHooks(self._record_usage)
+ try:
+ result = await Runner.run(
+ agent,
+ self.task,
+ max_turns=self.max_steps,
+ run_config=RunConfig(tracing_disabled=True),
+ hooks=hooks,
+ )
+ except Exception:
+ self.logger.exception("OpenAI MCP agent run failed")
+ raise
+ if self._total_input_tokens == 0 and self._total_output_tokens == 0:
+ for resp in result.raw_responses:
+ self._record_usage(resp.usage)
+ self.logger.info("OpenAI MCP agent run finished: %s", result)
+ return result
+ finally:
+ for state in logger_states:
+ if state:
+ restore_library_logger(*state)
+ if file_handler:
+ file_handler.flush()
+
+ def get_cost(self) -> UpdatableCostReport:
+ report = UpdatableCostReport.initialize_empty(model_name=self.model_id)
+ if self._total_input_tokens == 0 and self._total_output_tokens == 0:
+ return report
+
+ cost = litellm_tokens_cost(
+ model_name=self.model_id,
+ input_tokens=self._total_input_tokens,
+ output_tokens=self._total_output_tokens,
+ ).total_cost
+ report.add_cost(cost)
+ return report
+
+ def _build_prompt(self) -> str:
+ prompt = ""
+ if self.context:
+ prompt += f"Context: {self.context}\n\n"
+
+ prompt += (
+ "Complete this task using the available tools. Each tool corresponds to an action "
+ "you can take in the environment. Do not respond or ask clarification questions "
+ "unless done through a dedicated tool, and only if such tool exist. "
+ "Any plain message that is not a tool call will end the run in failure.\n"
+ )
+
+ if self.initial_observation is not None and not self.initial_observation.is_empty():
+ text = str(self.initial_observation).strip()
+ if text:
+ prompt += f"\nFirst Observation: {text}\n"
+
+ return prompt
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/openai/openai_mcp_agent.py b/labs/AgentStream/exgentic/src/exgentic/agents/openai/openai_mcp_agent.py
new file mode 100644
index 00000000..de508467
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/openai/openai_mcp_agent.py
@@ -0,0 +1,68 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from __future__ import annotations
+
+from typing import Any, ClassVar
+
+from pydantic import BaseModel, ConfigDict
+
+from ...core.agent import Agent
+from ...core.types import ModelSettings
+
+
+class MCPConfig(BaseModel):
+ model_config = ConfigDict(arbitrary_types_allowed=True)
+
+ client_session_timeout_seconds: float | None = None
+ http_timeout_seconds: float | None = None
+ sse_read_timeout_seconds: float | None = None
+ http_connect_timeout_seconds: float | None = None
+ headers: dict[str, str] | None = None
+ terminate_on_close: bool = True
+ max_retry_attempts: int = -1
+ retry_backoff_seconds_base: float = 1.0
+ cache_tools_list: bool = False
+ use_structured_content: bool = False
+ skip_health_check: bool = False
+ name: str | None = None
+ message_handler: Any | None = None
+
+
+class OpenAIMCPAgent(Agent):
+ display_name: ClassVar[str] = "OpenAI Solo"
+ slug_name: ClassVar[str] = "openai_solo"
+
+ model: str
+ max_steps: int = 150
+ model_settings: ModelSettings | None = None
+ mcp_config: MCPConfig | dict | None = None
+
+ @classmethod
+ def _get_instance_class(cls):
+ from .instance import OpenAIMCPAgentInstance
+
+ return OpenAIMCPAgentInstance
+
+ @classmethod
+ def _get_instance_class_ref(cls) -> str:
+ return "exgentic.agents.openai.instance:OpenAIMCPAgentInstance"
+
+ def _get_instance_kwargs(
+ self,
+ session_id: str,
+ ) -> dict[str, Any]:
+ return {
+ "session_id": session_id,
+ "model_id": self.model,
+ "max_steps": self.max_steps,
+ "model_settings": self.model_settings,
+ "mcp_config": self.mcp_config,
+ }
+
+ @property
+ def model_name(self) -> str: # type: ignore[override]
+ return str(self.model).split("/")[-1]
+
+ def get_models_names(self) -> list[str]: # type: ignore[override]
+ return [str(self.model)]
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/openai/requirements.txt b/labs/AgentStream/exgentic/src/exgentic/agents/openai/requirements.txt
new file mode 100644
index 00000000..6220519c
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/openai/requirements.txt
@@ -0,0 +1 @@
+openai-agents[litellm]
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/__init__.py b/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/evaluator.py b/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/evaluator.py
new file mode 100644
index 00000000..b49d1c1a
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/evaluator.py
@@ -0,0 +1,72 @@
+# Copyright 2026 The Google Research Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Modifications Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import logging
+
+logger = logging.getLogger(__name__)
+
+
+class TrajectoryEvaluator:
+
+ def __init__(self, llm_call_fn):
+ self._llm_call = llm_call_fn
+
+ def evaluate(
+ self,
+ intent: str,
+ think_list: list[str],
+ action_list: list[str],
+ observation_list: list[str],
+ final_response: str = "",
+ ) -> dict[str, str]:
+ from .prompts.eval_prompts import build_text_eval_prompt, extract_content
+
+ action_history = ""
+ for idx, act in enumerate(action_list):
+ think = think_list[idx] if idx < len(think_list) else ""
+ if think:
+ action_history += f"{idx+1}: {think}\n {act}\n"
+ else:
+ action_history += f"{idx+1}: {act}\n"
+
+ last_obs = observation_list[-5:] if len(observation_list) >= 5 else observation_list
+ combined_obs = "\n\n---\n\n".join(
+ f"[Page state {i+1}/{len(last_obs)}]\n{c}"
+ for i, c in enumerate(last_obs)
+ )
+
+ MAX_OBS_CHARS = 40000
+ if len(combined_obs) > MAX_OBS_CHARS:
+ combined_obs = combined_obs[:MAX_OBS_CHARS]
+
+ prompt, sys_msg = build_text_eval_prompt(
+ combined_obs, intent, final_response, action_history
+ )
+
+ msg_str = self._llm_call(prompt, sys_msg)
+
+ thoughts = extract_content(msg_str, "Thoughts:")
+ status_raw = extract_content(msg_str, "Status:").replace('"', "").strip().lower()
+
+
+ if "success" in status_raw:
+ status = "success"
+ else:
+ status = "failure"
+
+ return {"thoughts": thoughts, "status": status}
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/induce_memory.py b/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/induce_memory.py
new file mode 100644
index 00000000..3a0a94b2
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/induce_memory.py
@@ -0,0 +1,70 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Modifications Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import logging
+from typing import Callable
+
+logger = logging.getLogger(__name__)
+
+
+def format_trajectory(think_list: list[str], action_list: list[str], observation_list: list[str] | None = None) -> str:
+ trajectory = []
+ obs_list = observation_list or []
+ for i, (t, a) in enumerate(zip(think_list, action_list)):
+ if t:
+ trajectory.append(f"\n{t}\n\n\n{a}\n")
+ else:
+ obs = obs_list[i] if i < len(obs_list) else ""
+ if obs:
+ trajectory.append(f"\n{obs}\n\n\n{a}\n")
+ else:
+ trajectory.append(f"\n{a}\n")
+ return "\n\n".join(trajectory)
+
+
+def induce_memory(
+ query: str,
+ think_list: list[str],
+ action_list: list[str],
+ status: str,
+ eval_thoughts: str,
+ llm_call_fn: Callable[[str, str], str],
+ observation_list: list[str] | None = None,
+) -> list[str]:
+ from .prompts.memory_instruction import FAILED_SI, SUCCESSFUL_SI
+
+ trajectory = format_trajectory(think_list, action_list, observation_list)
+ trajectory = f"**Query:** {query}\n\n**Trajectory:**\n{trajectory}"
+
+ if eval_thoughts:
+ status_label = "succeeded" if status == "success" else "failed"
+ trajectory += f"\n\nThe task {status_label} because: {eval_thoughts}"
+
+ if status == "success":
+ generated_text = llm_call_fn(trajectory, SUCCESSFUL_SI)
+ else:
+ generated_text = llm_call_fn(trajectory, FAILED_SI)
+
+ memory_items = [item.strip() for item in generated_text.split("\n\n") if item.strip()]
+
+ logger.info(
+ "ReasoningBank memory induction: status=%s, generated %d items",
+ status, len(memory_items),
+ )
+
+ return memory_items
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/memory_management.py b/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/memory_management.py
new file mode 100644
index 00000000..1a622551
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/memory_management.py
@@ -0,0 +1,112 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Modifications Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import logging
+from typing import TYPE_CHECKING
+
+import numpy as np
+from sentence_transformers import SentenceTransformer
+
+if TYPE_CHECKING:
+ from .rb_store import MemoryEntry, ReasoningBankStore
+
+logger = logging.getLogger(__name__)
+
+RETRIEVAL_INSTRUCTION = (
+ "Given the prior web navigation queries, your task is to analyze a current "
+ "query's intent and select relevant prior queries that could help resolve it."
+)
+
+_st_model = None
+_st_model_name = None
+
+
+def _get_st_model(model_name: str = "all-MiniLM-L6-v2") -> SentenceTransformer:
+ global _st_model, _st_model_name
+ if _st_model is None or _st_model_name != model_name:
+ logger.info("Loading SentenceTransformer model: %s", model_name)
+ _st_model = SentenceTransformer(model_name)
+ _st_model_name = model_name
+ return _st_model
+
+
+def get_detailed_instruct(task_description: str, query: str) -> str:
+ return f"Instruct: {task_description}\nQuery: {query}"
+
+
+def l2_normalize(x: np.ndarray, axis: int = -1) -> np.ndarray:
+ norm = np.linalg.norm(x, axis=axis, keepdims=True)
+ norm = np.where(norm == 0, 1.0, norm)
+ return x / norm
+
+
+def compute_embedding(text: str, model: str = "all-MiniLM-L6-v2") -> list[float]:
+ st = _get_st_model(model)
+ vec = st.encode([text])[0]
+ return vec.tolist()
+
+
+def select_memory(
+ store: "ReasoningBankStore",
+ cur_query: str,
+ embedding_model: str,
+ top_k: int = 1,
+ exclude_task_id: str | None = None,
+) -> list["MemoryEntry"]:
+
+ cache_emb = store.get_embeddings_array()
+ if cache_emb is None or len(cache_emb) == 0:
+ logger.info("ReasoningBank retrieval: no cached embeddings, returning empty.")
+ return []
+
+ entries = store.get_entries()
+ entry_ids = store.get_entry_ids()
+
+ instruction_query = get_detailed_instruct(RETRIEVAL_INSTRUCTION, cur_query)
+ instruct_vec = np.array(
+ compute_embedding(instruction_query, embedding_model),
+ dtype=np.float32,
+ ).reshape(1, -1)
+
+ instruct_vec = l2_normalize(instruct_vec, axis=1)
+ cache_emb_norm = l2_normalize(cache_emb, axis=1)
+
+ scores = (instruct_vec @ cache_emb_norm.T).squeeze(0) * 100.0 # (N,)
+
+ id_score_pairs = []
+ for i, (eid, score) in enumerate(zip(entry_ids, scores)):
+ if exclude_task_id and eid == exclude_task_id:
+ continue
+ id_score_pairs.append((i, float(score)))
+
+ id_score_pairs.sort(key=lambda x: x[1], reverse=True)
+
+ top_entries = []
+ for idx, _score in id_score_pairs[:top_k]:
+ top_entries.append(entries[idx])
+
+ return top_entries
+
+
+def format_memories_for_prompt(entries: list["MemoryEntry"]) -> str:
+ mem_items = []
+ for entry in entries:
+ for item in entry.memory_items:
+ if item.strip():
+ mem_items.append(item.strip())
+ return "\n\n".join(mem_items)
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/prompts/__init__.py b/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/prompts/__init__.py
new file mode 100644
index 00000000..e69de29b
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/prompts/eval_prompts.py b/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/prompts/eval_prompts.py
new file mode 100644
index 00000000..ca4952cd
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/prompts/eval_prompts.py
@@ -0,0 +1,55 @@
+# Copyright 2026 The Google Research Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Modifications Copyright (C) 2026, The AgentStream organization and its contributors.
+
+
+def extract_content(text: str, start_tag: str) -> str:
+ for line in text.split("\n"):
+ if line.startswith(start_tag):
+ return line[len(start_tag):].strip()
+ return ""
+
+
+def build_text_eval_prompt(
+ cap: str, intent: str, response: str, last_actions: str
+) -> tuple[str, str]:
+ system_msg = """You are an expert in evaluating the performance of a task-solving agent. The agent is designed to help a human user complete a task by taking actions in an environment. Given the user's intent, the agent's action history, the environment's feedback, and the agent's response to the user, your goal is to decide whether the agent's execution is successful or not.
+
+*Strictness rules*
+Before calling a task successful, verify all three:
+- Completeness: every constraint in the intent is satisfied.
+- Grounding: every value or result the agent reports is traceable to a specific observation from the environment; values that were inferred, guessed, or summarized without a visible source count as failures.
+- Right target: when the task names a specific entity, confirm the agent acted on that exact entity and not an adjacent one.
+When uncertain on any of these, mark failure. A false success is more harmful than a false failure, because memory induction amplifies it into future behavior.
+
+*IMPORTANT*
+Format your response into two lines as shown below:
+
+Thoughts: "
+Status: "success" or "failure"
+"""
+ prompt = f"""User Intent: {intent}
+
+Action History:
+{last_actions}
+
+Environment feedback (last observations):
+
+```
+{cap}
+```
+
+Agent response to the user: {response if response else "N/A"}."""
+ return prompt, system_msg
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/prompts/memory_instruction.py b/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/prompts/memory_instruction.py
new file mode 100644
index 00000000..5b54a634
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/prompts/memory_instruction.py
@@ -0,0 +1,70 @@
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Modifications Copyright (C) 2026, The AgentStream organization and its contributors.
+
+SUCCESSFUL_SI = """
+You are an expert at analyzing agent task execution. You will be given a user query, the corresponding trajectory that represents **how an agent successfully accomplished the task**.
+
+## Guidelines
+You need to extract and summarize useful insights in the format of memory items based on the agent's successful trajectory.
+The goal of summarized memory items is to be helpful and generalizable for future similar tasks.
+
+## Important notes
+ - You must first think why the trajectory is successful, and then summarize the insights.
+ - You can extract *at most 3* memory items from the trajectory.
+ - You must not repeat similar or overlapping items.
+ - Prefer concrete, actionable procedures over abstract principles. Do not embed specific product names, queries, or literal string contents from the task.
+
+## Output Format
+Your output must strictly follow the Markdown format shown below:
+
+```
+# Memory Item i
+## Title
+## Description
+## Content <1-3 sentences describing the insights learned to successfully accomplishing similar tasks in the future>
+```
+"""
+
+FAILED_SI = """
+You are an expert at analyzing agent task execution. You will be given a user query, the corresponding trajectory that represents **how an agent attempted to resolve the task but failed**.
+
+## Guidelines
+You need to extract and summarize useful insights in the format of memory items based on the agent's failed trajectory.
+The goal of summarized memory items is to be helpful and generalizable for future similar tasks.
+
+## Important notes
+ - You must first reflect and think why the trajectory failed, and then summarize what lessons you have learned or strategies to prevent the failure in the future.
+ - You can extract *at most 3* memory items from the trajectory.
+ - You must not repeat similar or overlapping items.
+ - Prefer concrete, actionable recovery procedures over abstract principles. Do not embed specific product names, queries, or literal string contents from the task.
+
+## Output Format
+Your output must strictly follow the Markdown format shown below:
+
+```
+# Memory Item i
+## Title
+## Description
+## Content <1-3 sentences describing the insights learned to avoid such failures and successfully accomplishing similar tasks in the future>
+```
+"""
+
+MEMORY_INJECTION_INSTRUCTION = (
+ "Below are some memory items that I accumulated from past interaction from "
+ "the environment that may be helpful to solve the task. You can use it when "
+ "you feel it's relevant. In each step, please first explicitly discuss if "
+ "you want to use each memory item or not, and then take action."
+)
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/rb_agent.py b/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/rb_agent.py
new file mode 100644
index 00000000..ca3c1e15
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/rb_agent.py
@@ -0,0 +1,77 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+from typing import Any, ClassVar, Optional
+
+from pydantic import ConfigDict
+
+from ...core.agent import Agent
+from ...core.types import ModelSettings
+from ...utils.settings import RunnerName
+
+
+class ReasoningBankAgent(Agent):
+
+ display_name: ClassVar[str] = "ReasoningBank Agent"
+ slug_name: ClassVar[str] = "reasoning_bank"
+
+ model_config = ConfigDict(arbitrary_types_allowed=True)
+
+ model: str = "gpt-4o"
+ memory_model: Optional[str] = None
+ eval_model: Optional[str] = None
+ embedding_model: str = "all-MiniLM-L6-v2"
+
+ top_k_memories: int = 1
+ max_memory_items: int = 3
+
+ shuffle_mode: str = "isolated"
+
+ benchmark_id: Optional[str] = None
+
+ enable_tool_shortlisting: bool = False
+ max_selected_tools: int = 30
+
+ runner: RunnerName | None = None
+ model_settings: ModelSettings | None = None
+
+ @classmethod
+ def _get_instance_class(cls):
+ from .rb_instance import ReasoningBankAgentInstance
+ return ReasoningBankAgentInstance
+
+ @classmethod
+ def _get_instance_class_ref(cls) -> str:
+ return "exgentic.agents.reasoning_bank.rb_instance:ReasoningBankAgentInstance"
+
+ def _get_instance_kwargs(self, session_id: str) -> dict[str, Any]:
+ return {
+ "session_id": session_id,
+ "model": self.model,
+ "memory_model": self.memory_model or self.model,
+ "eval_model": self.eval_model or self.model,
+ "embedding_model": self.embedding_model,
+ "top_k_memories": self.top_k_memories,
+ "max_memory_items": self.max_memory_items,
+ "shuffle_mode": self.shuffle_mode,
+ "model_settings": self.model_settings,
+ "benchmark_id": self.benchmark_id,
+ "enable_tool_shortlisting": self.enable_tool_shortlisting,
+ "max_selected_tools": self.max_selected_tools,
+ }
+
+ @property
+ def model_name(self) -> str:
+ return str(self.model).split("/")[-1]
+
+ def get_models_names(self) -> list[str]:
+ names = [str(self.model)]
+ mm = self.memory_model or self.model
+ if mm != self.model:
+ names.append(str(mm))
+ em = self.eval_model or self.model
+ if em != self.model and em != mm:
+ names.append(str(em))
+ return names
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/rb_instance.py b/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/rb_instance.py
new file mode 100644
index 00000000..523a4f4a
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/rb_instance.py
@@ -0,0 +1,552 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import json
+import logging
+import time
+from typing import Any, Dict, Optional, Union
+
+import litellm
+from litellm import (
+ ChatCompletionAssistantMessage,
+ ChatCompletionSystemMessage,
+ ChatCompletionToolMessage,
+ ChatCompletionUserMessage,
+)
+
+from ...core.agent_instance import AgentInstance
+from ...core.types import (
+ Action,
+ ActionType,
+ Message,
+ MessageAction,
+ MessageObservation,
+ MessagePayload,
+ ModelSettings,
+ Observation,
+)
+from ...utils.cost import LiteLLMCostReport
+from ...utils.settings import get_settings
+
+from .rb_store import MemoryEntry, ReasoningBankStore
+from .memory_management import (
+ compute_embedding,
+ format_memories_for_prompt,
+ select_memory,
+)
+from .prompts.memory_instruction import MEMORY_INJECTION_INSTRUCTION
+from ..tool_shortlisting import shortlist_tools
+
+try:
+ from ...agents.litellm_tool_calling.utils import ToolCall, ToolsActionsRegistry
+except ImportError:
+ ToolsActionsRegistry = None
+ ToolCall = dict
+
+settings = get_settings()
+logger = logging.getLogger(__name__)
+
+
+class ReasoningBankAgentInstance(AgentInstance):
+
+ def __init__(
+ self,
+ session_id: str,
+ model: str = "gpt-4o",
+ memory_model: str = "gpt-4o",
+ eval_model: str = "gpt-4o",
+ embedding_model: str = "all-MiniLM-L6-v2",
+ top_k_memories: int = 1,
+ max_memory_items: int = 3,
+ shuffle_mode: str = "isolated",
+ model_settings: Optional[ModelSettings] = None,
+ benchmark_id: Optional[str] = None,
+ enable_tool_shortlisting: bool = False,
+ max_selected_tools: int = 30,
+ ) -> None:
+ super().__init__(session_id)
+
+ self.model = model
+ self.memory_model = memory_model
+ self.eval_model = eval_model
+ self.embedding_model = embedding_model
+ self.top_k_memories = top_k_memories
+ self.max_memory_items = max_memory_items
+ self.shuffle_mode = shuffle_mode
+ self.benchmark_id = benchmark_id
+ self.enable_tool_shortlisting = enable_tool_shortlisting
+ self.max_selected_tools = max_selected_tools
+
+ if model_settings is None:
+ self._model_settings = ModelSettings()
+ elif isinstance(model_settings, ModelSettings):
+ self._model_settings = model_settings
+ else:
+ self._model_settings = ModelSettings()
+
+ self._cost = LiteLLMCostReport.initialize_empty(model_name=self.model)
+ self._store: Optional[ReasoningBankStore] = None
+
+ self.messages: list[
+ Union[
+ ChatCompletionAssistantMessage,
+ ChatCompletionToolMessage,
+ ChatCompletionUserMessage,
+ ChatCompletionSystemMessage,
+ ]
+ ] = []
+
+ self._registry: Optional[Any] = None
+ self._all_actions: list[ActionType] = []
+ self._step_count: int = 0
+
+ self._think_list: list[str] = []
+ self._action_list: list[str] = []
+ self._observation_log: list[str] = []
+ self._action_log: list[Dict[str, Any]] = []
+
+ self._query_embedding: Optional[list[float]] = None
+ self._task_query: str = ""
+
+ def start(self, task: str, context: Dict[str, Any], actions: list[ActionType]) -> None:
+ super().start(task, context, actions)
+
+ self._all_actions = list(self.actions)
+ if ToolsActionsRegistry is not None:
+ self._registry = ToolsActionsRegistry(self._all_actions)
+ else:
+ self._registry = None
+
+ self._store = ReasoningBankStore.get_or_create(
+ shuffle_mode=self.shuffle_mode,
+ benchmark_id=self.benchmark_id,
+ )
+ self._store.increment_session()
+
+ self._task_query = task
+ if context:
+ context_str = ""
+ for k, v in context.items():
+ if isinstance(v, dict) and v.get("type") == "image_url":
+ continue
+ context_str += f"\n{k}: {v}"
+ self._task_query = f"{task}{context_str}"
+
+ try:
+ self._query_embedding = compute_embedding(
+ self._task_query, self.embedding_model
+ )
+ except Exception as e:
+ logger.warning("Failed to compute query embedding: %s", e)
+ self._query_embedding = None
+
+ retrieved_memories: list[MemoryEntry] = []
+ if self._query_embedding is not None and self._store.entry_count > 0:
+ try:
+ retrieved_memories = select_memory(
+ store=self._store,
+ cur_query=self._task_query,
+ embedding_model=self.embedding_model,
+ top_k=self.top_k_memories,
+ )
+ except Exception as e:
+ logger.warning("Memory retrieval failed: %s", e)
+
+ sys_prompt = "You are an expert agent that completes tasks using available tools."
+ if retrieved_memories:
+ memory_text = format_memories_for_prompt(retrieved_memories)
+ if memory_text.strip():
+ sys_prompt += "\n\n" + MEMORY_INJECTION_INSTRUCTION
+ sys_prompt += "\n\n" + memory_text
+
+ self.messages = [
+ ChatCompletionSystemMessage(role="system", content=sys_prompt),
+ ]
+
+ content_parts: list[Any] = []
+ ctx_str = ""
+ if self.context:
+ for k, v in self.context.items():
+ if isinstance(v, dict) and v.get("type") == "image_url":
+ content_parts.append({"type": "image_url", "image_url": {"url": v["data"], "detail": "high"}})
+ else:
+ ctx_str += f"\n<{k}>\n{v}\n"
+
+ text_content = f"{self.task}\n{ctx_str}"
+ if content_parts:
+ content_parts.insert(0, {"type": "text", "text": text_content})
+ self.messages.append(ChatCompletionUserMessage(role="user", content=content_parts))
+ else:
+ self.messages.append(
+ ChatCompletionUserMessage(role="user", content=text_content)
+ )
+
+ logger.info(
+ "ReasoningBank start: retrieved %d memories for task (store has %d entries)",
+ len(retrieved_memories), self._store.entry_count,
+ )
+
+ def react(self, observation: Optional[Observation]) -> Optional[Action]:
+ self._step_count += 1
+
+ self._observe(observation)
+
+ tools = self._assistant_tools()
+
+ response = self._completion(
+ model=self.model,
+ messages=self.messages,
+ tools=tools if tools else None,
+ )
+ if response is None:
+ return None
+
+ if response.usage:
+ self._cost.update_cost_from_tokens(
+ response.usage.prompt_tokens,
+ response.usage.completion_tokens,
+ )
+
+ choice = response["choices"][0]
+ message = choice["message"]
+ finish_reason = choice.get("finish_reason")
+
+ if finish_reason == "tool_calls" and self._registry is not None:
+ tool_calls = self._extract_tool_calls(message)
+ self.messages.append(
+ ChatCompletionAssistantMessage(
+ role="assistant",
+ tool_calls=[
+ {
+ "id": tc["id"],
+ "type": "function",
+ "function": {
+ "name": tc["name"],
+ "arguments": tc["arguments"],
+ },
+ }
+ for tc in tool_calls
+ ],
+ )
+ )
+
+ think = message.content if message.content else ""
+ self._think_list.append(think)
+ for tc in tool_calls:
+ self._action_list.append(f"{tc['name']}({tc['arguments']})")
+ self._action_log.append({
+ "step": self._step_count,
+ "action": tc["name"],
+ "arguments": tc["arguments"],
+ })
+
+ actions = self._registry.tool_calls_to_action(tool_calls)
+ return actions
+ else:
+ content = message.content if message.content else ""
+
+ if not content:
+ logger.warning(
+ "ReasoningBank step %d: empty content response (finish_reason=%s), "
+ "treating as agent inability to continue",
+ self._step_count, finish_reason,
+ )
+ return None
+
+ self.messages.append(
+ ChatCompletionAssistantMessage(role="assistant", content=content)
+ )
+
+ self._think_list.append(content)
+ self._action_list.append(f"send_msg_to_user('{content[:200]}')")
+
+ return MessageAction(arguments=Message(content=content))
+
+ def close(self) -> None:
+ if not self._store:
+ return
+
+ if not self._action_list:
+ logger.info("ReasoningBank close: no actions recorded, skipping memory induction.")
+ self._save_session_artifacts()
+ return
+
+ if self._query_embedding is None:
+ logger.info("ReasoningBank close: embedding unavailable, skipping memory induction.")
+ self._save_session_artifacts()
+ return
+
+ try:
+ self._run_post_session_learning()
+ except Exception as e:
+ logger.warning("ReasoningBank close: memory induction failed: %s", e)
+
+ self._save_session_artifacts()
+
+ def _save_session_artifacts(self) -> None:
+ if not self._store:
+ return
+ try:
+ cp = str(self.paths.agent_dir / "memory_checkpoint.json")
+ self._store.save_checkpoint(cp)
+ mt = str(self.paths.agent_dir / "memories.txt")
+ self._store.save_memories_text(mt)
+ except Exception as exc:
+ logger.warning("ReasoningBank: failed to save session artifacts: %s", exc)
+
+ def get_cost(self) -> LiteLLMCostReport:
+ return self._cost
+
+ def _run_post_session_learning(self) -> None:
+ from .evaluator import TrajectoryEvaluator
+ from .induce_memory import induce_memory
+
+ evaluator = TrajectoryEvaluator(
+ llm_call_fn=lambda prompt, sys_msg: self._llm_call_simple(
+ self.eval_model, prompt, system_msg=sys_msg
+ )
+ )
+
+ final_response = ""
+ for act in reversed(self._action_list):
+ if "send_msg_to_user" in act:
+ try:
+ final_response = act[act.index("(") + 1:act.rindex(")")]
+ final_response = final_response.strip("'\"")
+ except (ValueError, IndexError):
+ pass
+ break
+
+ eval_result = evaluator.evaluate(
+ intent=self.task or self._task_query,
+ think_list=self._think_list,
+ action_list=self._action_list,
+ observation_list=self._observation_log,
+ final_response=final_response,
+ )
+
+ status = "success" if eval_result["status"] == "success" else "fail"
+ eval_thoughts = eval_result.get("thoughts", "")
+
+ logger.info(
+ "ReasoningBank eval: status=%s, thoughts=%s",
+ status, eval_thoughts[:100],
+ )
+
+ memory_items = induce_memory(
+ query=self.task or self._task_query,
+ think_list=self._think_list,
+ action_list=self._action_list,
+ status=status,
+ eval_thoughts=eval_thoughts,
+ llm_call_fn=lambda user_msg, sys_msg: self._llm_call_simple(
+ self.memory_model, user_msg, system_msg=sys_msg
+ ),
+ observation_list=self._observation_log,
+ )
+
+ if memory_items and self._query_embedding is not None:
+ entry = MemoryEntry(
+ task_id=self.session_id,
+ query=self.task or self._task_query,
+ think_list=self._think_list,
+ action_list=self._action_list,
+ status=status,
+ memory_items=memory_items,
+ template_id=self.context.get("template_id") if self.context else None,
+ )
+ self._store.add_entry(entry, self._query_embedding)
+ logger.info(
+ "ReasoningBank: stored %d memory items (store now has %d entries)",
+ len(memory_items), self._store.entry_count,
+ )
+
+ def _completion(self, **kwargs) -> Any:
+ """Standard completion with tool support and retry logic."""
+ call_kwargs = self._model_settings.model_dump(
+ exclude_none=True,
+ exclude={"num_retries", "retry_after", "retry_strategy"},
+ )
+ call_kwargs.update(kwargs)
+ if call_kwargs.get("tools") is None:
+ call_kwargs.pop("tools", None)
+
+ max_attempts = 3
+ for attempt in range(max_attempts):
+ try:
+ response = litellm.completion(**call_kwargs)
+ choice = response["choices"][0] if response.get("choices") else None
+ if choice:
+ msg = choice.get("message") or {}
+ has_content = bool(msg.get("content"))
+ has_tools = bool(msg.get("tool_calls"))
+ if not has_content and not has_tools:
+ if attempt + 1 < max_attempts:
+ logger.warning(
+ "ReasoningBank LLM call attempt %d/%d: empty response "
+ "(finish_reason=%s), retrying...",
+ attempt + 1, max_attempts,
+ choice.get("finish_reason"),
+ )
+ time.sleep(2 ** attempt)
+ continue
+ return response
+ except Exception as exc:
+ logger.warning(
+ "ReasoningBank LLM call attempt %d/%d failed: %s",
+ attempt + 1, max_attempts, exc,
+ )
+ if attempt + 1 >= max_attempts:
+ raise
+ time.sleep(2 ** attempt)
+ return None
+
+ def _llm_call_simple(
+ self,
+ model: str,
+ prompt: str,
+ *,
+ system_msg: str = "",
+ json_mode: bool = False,
+ ) -> str:
+ messages: list = []
+ if system_msg:
+ messages.append({"role": "system", "content": system_msg})
+ messages.append({"role": "user", "content": prompt})
+
+ kwargs: Dict[str, Any] = {
+ "model": model,
+ "messages": messages,
+ "temperature": 1.0,
+ }
+ if json_mode:
+ kwargs["response_format"] = {"type": "json_object"}
+
+ max_attempts = 3
+ for attempt in range(max_attempts):
+ try:
+ resp = litellm.completion(**kwargs)
+ if resp.usage:
+ self._cost.update_cost_from_tokens(
+ resp.usage.prompt_tokens,
+ resp.usage.completion_tokens,
+ )
+ return resp.choices[0].message.content or ""
+ except Exception as exc:
+ if attempt + 1 >= max_attempts:
+ logger.error("Simple LLM call failed: %s", exc)
+ return ""
+ time.sleep(2 ** attempt)
+ return ""
+
+ def _observe(self, observation: Optional[Observation]) -> None:
+ if observation is None:
+ return
+
+ observations = observation.to_observation_list()
+ if observation.is_empty():
+ if not any(obs.invoking_actions for obs in observations):
+ return
+
+ for obs in observations:
+ if isinstance(obs, MessageObservation) and isinstance(
+ obs.result, MessagePayload
+ ):
+ self.messages.append(
+ ChatCompletionUserMessage(role="user", content=obs.result.message)
+ )
+ self._observation_log.append(obs.result.message)
+ continue
+
+ if len(obs.invoking_actions) > 0:
+ invoking = obs.invoking_actions[0]
+ if invoking.name == "message":
+ text = str(obs)
+ self.messages.append(
+ ChatCompletionUserMessage(role="user", content=text)
+ )
+ self._observation_log.append(text)
+ continue
+
+ tool_call_id = invoking.id
+ if not (
+ isinstance(tool_call_id, str)
+ and tool_call_id.startswith("call_")
+ ):
+ if self._registry is not None:
+ tool_call_id = (
+ self._registry.action_id_to_tool_call_id.get(
+ tool_call_id, tool_call_id
+ )
+ )
+
+ value = obs.result
+ try:
+ content = json.dumps(value, ensure_ascii=False, separators=(",", ":"))
+ except TypeError:
+ content = str(value)
+
+ if tool_call_id is not None:
+ self.messages.append(
+ ChatCompletionToolMessage(
+ role="tool",
+ tool_call_id=tool_call_id,
+ content=content,
+ )
+ )
+ else:
+ self.messages.append(
+ ChatCompletionUserMessage(
+ role="user",
+ content=f"Tool result: {content}",
+ )
+ )
+ self._observation_log.append(content)
+ else:
+ content = str(obs.result) if hasattr(obs, "result") else str(obs)
+ self.messages.append(
+ ChatCompletionUserMessage(role="user", content=content)
+ )
+ self._observation_log.append(content)
+
+
+ def _assistant_tools(self) -> list | None:
+ if self._registry is None:
+ return None
+ tools = self._registry.openai_tools()
+ if not tools:
+ return None
+
+ if not self.enable_tool_shortlisting:
+ return tools
+
+ def _cost_cb(usage):
+ if usage:
+ self._cost.update_cost_from_tokens(
+ usage.prompt_tokens, usage.completion_tokens
+ )
+
+ return shortlist_tools(
+ tools=tools,
+ max_selected=self.max_selected_tools,
+ messages=self.messages,
+ completion_fn=self._completion,
+ model=self.model,
+ logger=logger,
+ cost_callback=_cost_cb,
+ )
+
+ @staticmethod
+ def _extract_tool_calls(message: Any) -> list[dict[str, str]]:
+ if not hasattr(message, "tool_calls") or not message.tool_calls:
+ return []
+ tool_calls = []
+ for tc in message.tool_calls:
+ tool_calls.append({
+ "name": tc.function.name,
+ "arguments": tc.function.arguments,
+ "id": tc.id,
+ })
+ return tool_calls
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/rb_store.py b/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/rb_store.py
new file mode 100644
index 00000000..0d8c4ad3
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/reasoning_bank/rb_store.py
@@ -0,0 +1,164 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import json
+import logging
+import threading
+from dataclasses import asdict, dataclass
+from pathlib import Path
+from typing import Any, ClassVar
+
+import numpy as np
+
+logger = logging.getLogger(__name__)
+
+
+@dataclass
+class MemoryEntry:
+
+ task_id: str
+ query: str
+ think_list: list[str]
+ action_list: list[str]
+ status: str
+ memory_items: list[str]
+ template_id: str | None = None
+
+ def to_dict(self) -> dict[str, Any]:
+ return asdict(self)
+
+ @classmethod
+ def from_dict(cls, d: dict[str, Any]) -> "MemoryEntry":
+ return cls(
+ task_id=d["task_id"],
+ query=d["query"],
+ think_list=d.get("think_list", []),
+ action_list=d.get("action_list", []),
+ status=d["status"],
+ memory_items=d.get("memory_items", []),
+ template_id=d.get("template_id"),
+ )
+
+
+class ReasoningBankStore:
+
+ _instances: ClassVar[dict[str, "ReasoningBankStore"]] = {}
+ _class_lock: ClassVar[threading.Lock] = threading.Lock()
+
+ def __init__(self, store_id: str) -> None:
+ self._store_id = store_id
+ self._lock = threading.Lock()
+ self._entries: list[MemoryEntry] = []
+ self._embeddings: list[list[float]] = []
+ self._session_count: int = 0
+
+ @classmethod
+ def get_or_create(
+ cls,
+ shuffle_mode: str = "isolated",
+ benchmark_id: str | None = None,
+ ) -> "ReasoningBankStore":
+ if shuffle_mode == "isolated":
+ store_id = f"rb_isolated_{benchmark_id or 'default'}"
+ elif shuffle_mode == "sequential":
+ store_id = "rb_sequential_global"
+ elif shuffle_mode == "interleaved":
+ store_id = "rb_interleaved_global"
+ else:
+ store_id = f"rb_{shuffle_mode}_{benchmark_id or 'default'}"
+
+ with cls._class_lock:
+ if store_id not in cls._instances:
+ cls._instances[store_id] = cls(store_id)
+ return cls._instances[store_id]
+
+ @classmethod
+ def list_stores(cls) -> dict[str, "ReasoningBankStore"]:
+ with cls._class_lock:
+ return dict(cls._instances)
+
+ @classmethod
+ def reset_all(cls) -> None:
+ with cls._class_lock:
+ cls._instances.clear()
+
+ @property
+ def store_id(self) -> str:
+ return self._store_id
+
+ @property
+ def session_count(self) -> int:
+ with self._lock:
+ return self._session_count
+
+ @property
+ def entry_count(self) -> int:
+ with self._lock:
+ return len(self._entries)
+
+ def increment_session(self) -> int:
+ with self._lock:
+ self._session_count += 1
+ return self._session_count
+
+ def add_entry(self, entry: MemoryEntry, embedding: list[float]) -> None:
+ with self._lock:
+ self._entries.append(entry)
+ self._embeddings.append(embedding)
+
+ def get_entries(self) -> list[MemoryEntry]:
+ with self._lock:
+ return list(self._entries)
+
+ def get_embeddings_array(self) -> np.ndarray | None:
+ with self._lock:
+ if not self._embeddings:
+ return None
+ return np.array(self._embeddings, dtype=np.float32)
+
+ def get_entry_ids(self) -> list[str]:
+ with self._lock:
+ return [e.task_id for e in self._entries]
+
+ def save_checkpoint(self, path: str) -> None:
+ with self._lock:
+ data = {
+ "store_id": self._store_id,
+ "session_count": self._session_count,
+ "entries": [e.to_dict() for e in self._entries],
+ "embeddings": self._embeddings,
+ }
+ Path(path).parent.mkdir(parents=True, exist_ok=True)
+ with open(path, "w", encoding="utf-8") as f:
+ json.dump(data, f, ensure_ascii=False)
+ logger.info("ReasoningBankStore[%s]: saved checkpoint (%d entries) to %s",
+ self._store_id, len(self._entries), path)
+
+ def save_memories_text(self, path: str) -> None:
+ Path(path).parent.mkdir(parents=True, exist_ok=True)
+ with self._lock:
+ lines = [
+ f"# ReasoningBank Memory Store: {self._store_id}",
+ f"# Entries: {len(self._entries)}",
+ f"# Sessions: {self._session_count}",
+ "",
+ ]
+ for i, entry in enumerate(self._entries):
+ lines.append(f"--- Entry {i + 1} [{entry.task_id[:20]}] status={entry.status} ---")
+ for item in entry.memory_items:
+ lines.append(item.strip())
+ lines.append("")
+ with open(path, "w", encoding="utf-8") as f:
+ f.write("\n".join(lines))
+
+ def load_checkpoint(self, path: str) -> None:
+ with open(path, "r", encoding="utf-8") as f:
+ data = json.load(f)
+ with self._lock:
+ self._session_count = data.get("session_count", 0)
+ self._entries = [MemoryEntry.from_dict(d) for d in data.get("entries", [])]
+ self._embeddings = data.get("embeddings", [])
+ logger.info("ReasoningBankStore[%s]: loaded checkpoint (%d entries) from %s",
+ self._store_id, len(self._entries), path)
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/replay/__init__.py b/labs/AgentStream/exgentic/src/exgentic/agents/replay/__init__.py
new file mode 100644
index 00000000..367fec41
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/replay/__init__.py
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/replay/replay_agent.py b/labs/AgentStream/exgentic/src/exgentic/agents/replay/replay_agent.py
new file mode 100644
index 00000000..ab8d5797
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/replay/replay_agent.py
@@ -0,0 +1,107 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+"""ReplayAgent — replays a recorded trajectory against a benchmark.
+
+Usage::
+
+ evaluate(benchmark="gsm8k", agent="replay", agent_kwargs={"recording": "path/to/recording"})
+
+A *recording* is a directory containing:
+ trajectory.jsonl — the recorded action/observation events
+ session.json — session manifest (task, context, actions schema)
+
+These files are produced automatically by ``exgentic evaluate`` (under
+``outputs//sessions//``).
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any, ClassVar
+
+from ...core.agent import Agent
+from ...core.agent_instance import AgentInstance
+from ...core.types import Action, Observation
+
+
+class ReplayAgentInstance(AgentInstance):
+ """Replays recorded actions from a trajectory file."""
+
+ def __init__(
+ self,
+ *,
+ session_id: str,
+ trajectory_path: Path,
+ ) -> None:
+ super().__init__(session_id=session_id)
+ self._actions = self._load_actions(trajectory_path)
+ self._step = 0
+
+ def start(self, task, context, actions):
+ """Receive work payload and build the action_types lookup."""
+ super().start(task, context, actions)
+ self._action_types = {at.name: at for at in self.actions}
+
+ @staticmethod
+ def _load_actions(trajectory_path: Path) -> list[dict]:
+ """Extract action events from a trajectory JSONL file."""
+ actions = []
+ with open(trajectory_path, encoding="utf-8") as f:
+ for line in f:
+ line = line.strip()
+ if not line:
+ continue
+ event = json.loads(line)
+ if event.get("event") == "action":
+ actions.append(event["action"])
+ return actions
+
+ def react(self, observation: Observation | None) -> Action | None:
+ if self._step >= len(self._actions):
+ return None # No more recorded actions — signal done
+
+ action_data = self._actions[self._step]
+ self._step += 1
+
+ name = action_data.get("name", "")
+ arguments = action_data.get("arguments", {})
+
+ action_type = self._action_types.get(name)
+ if action_type is None:
+ from ...core.actions import build_unknown_action
+
+ return build_unknown_action(name, arguments)
+
+ return action_type.build_action(arguments)
+
+ def close(self) -> None:
+ pass
+
+
+class ReplayAgent(Agent):
+ """Agent that replays pre-recorded actions from a trajectory file."""
+
+ display_name: ClassVar[str] = "Replay Agent"
+ slug_name: ClassVar[str] = "replay"
+ recording: str # Path to the recording directory (or trajectory.jsonl file)
+ runner: str | None = "direct" # No external deps — run in host process
+
+ @classmethod
+ def _get_instance_class(cls):
+ return ReplayAgentInstance
+
+ def _get_instance_kwargs(
+ self,
+ session_id: str,
+ ) -> dict[str, Any]:
+ recording_path = Path(self.recording)
+ if recording_path.is_file():
+ trajectory_path = recording_path
+ else:
+ trajectory_path = recording_path / "trajectory.jsonl"
+ return {
+ "session_id": session_id,
+ "trajectory_path": trajectory_path,
+ }
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/replay/replay_benchmark.py b/labs/AgentStream/exgentic/src/exgentic/agents/replay/replay_benchmark.py
new file mode 100644
index 00000000..c9af05f9
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/replay/replay_benchmark.py
@@ -0,0 +1,92 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+"""ReplayBenchmark — replays a full recorded session (agent + environment).
+
+Pairs with ReplayAgent + ReplaySession to test the full execution loop
+without needing any benchmark dependencies installed.
+
+Usage (from tests)::
+
+ benchmark = ReplayBenchmark(recording_dir="path/to/recording")
+ agent = ReplayAgent(recording="path/to/recording")
+ results = evaluate(benchmark=benchmark, agent=agent)
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any, ClassVar
+
+from ...core.benchmark import Benchmark
+from ...core.evaluator import Evaluator
+from ...core.types import BenchmarkResults, SessionIndex
+from .replay_session import ReplaySession
+
+
+class ReplayEvaluator(Evaluator):
+ """Evaluator that returns session kwargs for ReplaySession."""
+
+ def __init__(self, recording_dir: str) -> None:
+ self._recording_dir = recording_dir
+
+ def list_tasks(self) -> list[str]:
+ recording = Path(self._recording_dir)
+ # Try to get task_id from session.json
+ manifest_path = recording / "session.json"
+ if manifest_path.exists():
+ manifest = json.loads(manifest_path.read_text())
+ task_id = manifest.get("task_id", "0")
+ return [str(task_id)]
+ return ["0"]
+
+ def get_session_kwargs(self, index: SessionIndex) -> dict[str, Any]:
+ return {
+ "recording_dir": self._recording_dir,
+ "session_id": index.session_id,
+ }
+
+ def aggregate_sessions(self, sessions: list[SessionIndex]) -> BenchmarkResults:
+ paths = self.get_sessions_paths(sessions)
+ scores = []
+ for p in paths:
+ results_path = p.benchmark_results
+ if results_path.exists():
+ data = json.loads(results_path.read_text())
+ score = data.get("score")
+ if score is not None:
+ scores.append(float(score))
+
+ avg_score = sum(scores) / len(scores) if scores else 0.0
+ return BenchmarkResults(
+ benchmark_name="replay",
+ total_tasks=len(sessions),
+ score=avg_score,
+ )
+
+
+class ReplayBenchmark(Benchmark):
+ """Benchmark that replays recorded sessions from a directory."""
+
+ display_name: ClassVar[str] = "Replay Benchmark"
+ slug_name: ClassVar[str] = "replay"
+ recording_dir: str
+
+ @classmethod
+ def _get_evaluator_class(cls):
+ return ReplayEvaluator
+
+ @classmethod
+ def _get_session_class(cls):
+ return ReplaySession
+
+ def _get_evaluator_kwargs(self) -> dict[str, Any]:
+ return {"recording_dir": self.recording_dir}
+
+ def runner_kwargs(self) -> dict[str, Any]:
+ kw = super().runner_kwargs()
+ if self.resolve_runner() == "docker":
+ recording_dir = str(Path(self.recording_dir).resolve())
+ kw.setdefault("volumes", {})[recording_dir] = recording_dir
+ return kw
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/replay/replay_session.py b/labs/AgentStream/exgentic/src/exgentic/agents/replay/replay_session.py
new file mode 100644
index 00000000..e13b38ee
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/replay/replay_session.py
@@ -0,0 +1,139 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+"""ReplaySession — replays recorded observations from a trajectory.
+
+Used together with ReplayAgent to test the full execution loop
+without needing any benchmark dependencies installed.
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any
+
+from json_schema_to_pydantic import create_model
+
+from ...core.session import Session
+from ...core.types import Action, ActionType, Observation, SessionScore, SingleAction, SingleObservation
+
+
+def _action_type_from_schema(entry: dict) -> ActionType:
+ """Reconstruct an ActionType from a session.json action entry."""
+ schema = entry["arguments_schema"]
+ args_model = create_model(schema)
+
+ # Create a SingleAction subclass with the right name + arguments type
+ action_cls = type(
+ f"{entry['name']}_Action",
+ (SingleAction,),
+ {"__annotations__": {"name": str, "arguments": args_model}},
+ )
+
+ return ActionType(
+ name=entry["name"],
+ description=entry.get("description", ""),
+ cls=action_cls,
+ is_finish=entry.get("is_finish", False),
+ is_message=entry.get("is_message", False),
+ is_hidden=entry.get("is_hidden", False),
+ )
+
+
+class ReplaySession(Session):
+ """Session that replays recorded observations from a trajectory file.
+
+ Does not require any benchmark dependencies — everything is
+ reconstructed from the recording (session.json + trajectory.jsonl).
+ """
+
+ def __init__(
+ self,
+ recording_dir: str,
+ *,
+ session_id: str | None = None,
+ ) -> None:
+ recording = Path(recording_dir)
+ manifest = json.loads((recording / "session.json").read_text())
+
+ self._task_id_val = manifest.get("task_id", "")
+ self._task_val = manifest.get("task", "")
+ self._context_val = manifest.get("context", {})
+ self._action_types = [_action_type_from_schema(a) for a in manifest.get("actions", [])]
+
+ # Load recorded observations and score from trajectory/results
+ self._observations: list[Any] = []
+ trajectory = recording / "trajectory.jsonl"
+ with open(trajectory, encoding="utf-8") as f:
+ for line in f:
+ line = line.strip()
+ if not line:
+ continue
+ event = json.loads(line)
+ if event.get("event") == "observation" and event.get("observation") is not None:
+ self._observations.append(event["observation"])
+
+ # Load recorded score
+ results_path = recording / "results.json"
+ if results_path.exists():
+ results = json.loads(results_path.read_text())
+ self._recorded_score = results.get("details", results)
+ else:
+ self._recorded_score = {"score": 0.0, "success": False, "is_finished": True}
+
+ self._step_idx = 0
+ self._done = False
+
+ if session_id is not None:
+ self._session_id = session_id
+
+ super().__init__()
+
+ @property
+ def task_id(self) -> str:
+ return self._task_id_val
+
+ @property
+ def task(self) -> str:
+ return self._task_val
+
+ @property
+ def context(self) -> dict[str, Any]:
+ return self._context_val
+
+ @property
+ def actions(self) -> list[ActionType]:
+ return self._action_types
+
+ def _next_observation(self) -> Observation | None:
+ if self._step_idx < len(self._observations):
+ obs = self._observations[self._step_idx]
+ self._step_idx += 1
+ return SingleObservation(result=obs.get("result") if isinstance(obs, dict) else obs)
+ return None
+
+ def start(self) -> Observation | None:
+ return self._next_observation()
+
+ def step(self, action: Action) -> Observation | None:
+ obs = self._next_observation()
+ if obs is None:
+ self._done = True
+ return obs
+
+ def done(self) -> bool:
+ return self._done
+
+ def score(self) -> SessionScore:
+ data = self._recorded_score
+ return SessionScore(
+ score=float(data.get("score", 0.0)),
+ success=bool(data.get("success", False)),
+ is_finished=data.get("is_finished", True),
+ session_metrics=data.get("session_metrics", {}),
+ session_metadata=data.get("session_metadata", {}),
+ )
+
+ def close(self) -> None:
+ pass
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/__init__.py b/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/__init__.py
new file mode 100644
index 00000000..882dc64e
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/__init__.py
@@ -0,0 +1,12 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from .base_agent import SmolagentBaseAgent
+from .code_agent import SmolagentCodeAgent
+from .tool_calling_agent import SmolagentToolCallingAgent
+
+__all__ = [
+ "SmolagentBaseAgent",
+ "SmolagentCodeAgent",
+ "SmolagentToolCallingAgent",
+]
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/base_agent.py b/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/base_agent.py
new file mode 100644
index 00000000..1a0634ec
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/base_agent.py
@@ -0,0 +1,36 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from typing import Any, ClassVar
+
+from ...core.agent import Agent
+from ...core.types import ModelSettings
+
+
+class SmolagentBaseAgent(Agent):
+ display_name: ClassVar[str] = "SmolAgents Base Agent"
+ slug_name: ClassVar[str] = "smolagents_base"
+
+ model: str = "watsonx/meta-llama/llama-3-3-70b-instruct"
+ max_steps: int = 150
+ model_settings: ModelSettings | None = None
+ retry_on_all_errors: bool = True
+
+ def _get_instance_kwargs(
+ self,
+ session_id: str,
+ ) -> dict[str, Any]:
+ return {
+ "session_id": session_id,
+ "model_id": self.model,
+ "max_steps": self.max_steps,
+ "model_settings": self.model_settings,
+ "retry_on_all_errors": self.retry_on_all_errors,
+ }
+
+ @property
+ def model_name(self) -> str: # type: ignore[override]
+ return str(self.model).split("/")[-1]
+
+ def get_models_names(self) -> list[str]: # type: ignore[override]
+ return [str(self.model)]
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/base_instance.py b/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/base_instance.py
new file mode 100644
index 00000000..8a1ce679
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/base_instance.py
@@ -0,0 +1,152 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+import functools
+import logging
+from abc import abstractmethod
+from collections.abc import Callable
+
+from rich.console import Console
+from smolagents import LiteLLMModel
+from smolagents.models import is_rate_limit_error
+from smolagents.monitoring import AgentLogger, LogLevel
+from smolagents.tools import Tool, tool
+from smolagents.utils import AgentError, Retrying
+
+from ...adapters.agents.code_agent import CodeAgentInstance
+from ...core.context import get_context
+from ...core.types import ModelSettings, RetryStrategy
+from ...integrations.litellm.health import check_model_accessible_sync
+from ...observers.logging import close_logger
+from ...utils.cost import CostReport, LiteLLMCostReport
+from ...utils.settings import get_settings
+
+settings = get_settings()
+
+
+class ContextInjectingLiteLLMModel(LiteLLMModel):
+ """Wrapper around LiteLLMModel that injects context into litellm_metadata."""
+
+ def generate(self, *args, **kwargs):
+ """Inject context into litellm_metadata before calling the model."""
+ # Use 'metadata' parameter instead of 'litellm_metadata'
+ # LiteLLM passes 'metadata' to callbacks in litellm_params.metadata
+ kwargs.setdefault("metadata", {})["context"] = get_context()
+
+ return super().generate(*args, **kwargs)
+
+
+class SmolagentBaseAgentInstance(CodeAgentInstance):
+ def __init__(
+ self,
+ session_id: str,
+ model_id: str,
+ max_steps: int = 150,
+ model_settings: ModelSettings | None = None,
+ retry_on_all_errors: bool = True,
+ ):
+ super().__init__(session_id)
+ self.model_id = model_id
+ self.max_steps = max_steps
+ if model_settings is None:
+ self.model_settings = ModelSettings()
+ elif isinstance(model_settings, ModelSettings):
+ self.model_settings = model_settings
+ else:
+ raise ValueError("model_settings must be a ModelSettings instance.")
+ self._retry_on_all_errors = retry_on_all_errors
+ self._agent = None
+ self._model = None
+
+ # Check model accessibility
+ check_model_accessible_sync(self.model_id, logger=self.logger)
+
+ def run_code_agent(self, functions: list[Callable]) -> None:
+ def _wrap_tool(fn: Callable) -> Callable:
+ @functools.wraps(fn)
+ def wrapper(*args, **kwargs):
+ try:
+ return fn(*args, **kwargs)
+ except RuntimeError as exc:
+ if "after close" in str(exc):
+ agent_logger = self.get_smolagent_logger()
+ if agent_logger is None:
+ agent_logger = AgentLogger(
+ console=Console(),
+ level=LogLevel.ERROR,
+ )
+ raise AgentError("Agent interrupted (session closed).", agent_logger) from exc
+ raise
+
+ return wrapper
+
+ tools = [tool(_wrap_tool(function)) for function in functions]
+ return self.run_smolagent(tools=tools)
+
+ def get_smolagent_logger(self):
+ smolagent_logger = None
+ for handler in self.logger.handlers:
+ if isinstance(handler, logging.FileHandler):
+ console = Console(
+ file=handler.stream,
+ force_terminal=False,
+ color_system=None,
+ highlight=False,
+ )
+ smolagent_logger = AgentLogger(console=console, level=LogLevel.DEBUG)
+ return smolagent_logger
+
+ def get_internal_model(self):
+ if self._model is None:
+ temperature = self.model_settings.temperature
+ self._model = ContextInjectingLiteLLMModel(
+ model_id=self.model_id,
+ temperature=temperature if temperature is not None else 1.0,
+ max_tokens=self.model_settings.max_tokens,
+ caching=settings.litellm_caching,
+ )
+ num_retries = self.model_settings.num_retries or 0
+ max_attempts = num_retries + 1 if num_retries > 0 else 1
+ retry_strategy = self.model_settings.retry_strategy.value
+ exponential_base = 2.0 if retry_strategy == RetryStrategy.EXPONENTIAL_BACKOFF.value else 1.0
+ log_level = logging._nameToLevel.get(settings.log_level, logging.INFO)
+ self._model.retryer = Retrying(
+ max_attempts=max_attempts,
+ wait_seconds=self.model_settings.retry_after,
+ exponential_base=exponential_base,
+ jitter=False,
+ retry_predicate=self.retry_predicate,
+ reraise=True,
+ before_sleep_logger=(self.logger, log_level),
+ after_logger=None,
+ )
+ return self._model
+
+ def retry_predicate(self, exc: BaseException) -> bool:
+ if self._retry_on_all_errors:
+ return True
+ return is_rate_limit_error(exc)
+
+ @abstractmethod
+ def run_smolagent(self, tools: list[Tool]):
+ raise NotImplementedError
+
+ def close(self):
+ self.logger.info("Interrupting Smolagent...")
+ if self._agent is not None:
+ self._agent.interrupt()
+ super().close()
+ self.logger.debug("Closing logger.")
+ close_logger(self.logger)
+
+ def get_cost(self) -> CostReport:
+ if self._agent is None:
+ return LiteLLMCostReport.initialize_empty(model_name=self.model_id)
+
+ token_usage = self._agent.monitor.get_total_token_counts()
+
+ return LiteLLMCostReport.from_token_counts(
+ model_name=self.model_id,
+ input_tokens=token_usage.input_tokens,
+ output_tokens=token_usage.output_tokens,
+ )
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/code_agent.py b/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/code_agent.py
new file mode 100644
index 00000000..9466f716
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/code_agent.py
@@ -0,0 +1,21 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from typing import ClassVar
+
+from .base_agent import SmolagentBaseAgent
+
+
+class SmolagentCodeAgent(SmolagentBaseAgent):
+ display_name: ClassVar[str] = "SmolAgents Code"
+ slug_name: ClassVar[str] = "smolagents_code"
+
+ @classmethod
+ def _get_instance_class(cls):
+ from .code_instance import SmolagentCodeAgentInstance
+
+ return SmolagentCodeAgentInstance
+
+ @classmethod
+ def _get_instance_class_ref(cls) -> str:
+ return "exgentic.agents.smolagents.code_instance:SmolagentCodeAgentInstance"
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/code_instance.py b/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/code_instance.py
new file mode 100644
index 00000000..5d52cb8c
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/code_instance.py
@@ -0,0 +1,58 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+import os
+
+import yaml
+from smolagents import CodeAgent as SmolagentBaseCodeAgent
+from smolagents.tools import Tool
+from smolagents.utils import AgentError
+
+from .base_instance import SmolagentBaseAgentInstance
+
+
+class SmolagentCodeAgentInstance(SmolagentBaseAgentInstance):
+ """Smolagent implementation."""
+
+ def run_smolagent(self, tools: list[Tool]):
+ # Load custom structured prompt templates from YAML next to this module
+ prompt_path = os.path.join(os.path.dirname(__file__), "structured_code_agent.yaml")
+ try:
+ with open(prompt_path, encoding="utf-8-sig") as f:
+ prompt_templates = yaml.safe_load(f)
+ except Exception:
+ prompt_templates = None
+
+ self._agent = SmolagentBaseCodeAgent(
+ tools=tools,
+ model=self.get_internal_model(),
+ prompt_templates=prompt_templates,
+ use_structured_outputs_internally=True,
+ logger=self.get_smolagent_logger(),
+ )
+ # Remove built-in final_answer; termination should happen by interacting with the benchmark (finish action).
+ self._agent.tools.pop("final_answer", None)
+
+ prompt = f"Task: {self.task}\n\n"
+ if self.context:
+ prompt += f"Context: {self.context}\n\n"
+ prompt += (
+ "Complete this task using the available functions. "
+ "Each function corresponds to an action you can take to solve the given task.\n"
+ "Every action should be taken only by calling one of the functions. "
+ "If one function fail, consider using another, at any given point one of the functions\n"
+ "can be a valid next step. At any point you should executing actions by writing code. "
+ "do not call tools with tool calling mechanism.\n\n"
+ "Printing or any other code will be visible only by you alone.\n\n"
+ # "Always provide parameter names when calling function. Do not rely on positional arguments.\n"
+ )
+ if self.initial_observation is not None and not self.initial_observation.is_empty():
+ text = str(self.initial_observation).strip()
+ if text:
+ prompt += f"\nFirst Observation: {text}\n"
+ try:
+ self._agent.run(task=prompt, max_steps=self.max_steps)
+
+ except AgentError as e:
+ self.logger.info(f"AgentError: {e}")
+ raise
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/requirements.txt b/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/requirements.txt
new file mode 100644
index 00000000..f7a97d41
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/requirements.txt
@@ -0,0 +1 @@
+smolagents>=1.13.0
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/structured_code_agent.yaml b/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/structured_code_agent.yaml
new file mode 100644
index 00000000..411dc2bb
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/structured_code_agent.yaml
@@ -0,0 +1,257 @@
+system_prompt: |-
+ You are an expert assistant who can solve any task using code blobs. You will be given a task to solve as best you can.
+ To do so, you have been given access to a list of tools: these tools are basically Python functions which you can call with code.
+ To solve the task, you must plan forward to proceed in a series of steps, in a cycle of 'Thought:', 'Code:', and 'Observation:' sequences.
+
+ At each step, in the 'Thought:' attribute, you should first explain your reasoning towards solving the task and the tools that you want to use.
+ Then in the 'Code' attribute, you should write the code in simple Python.
+ During each intermediate step, you can use 'print()' to save whatever important information you will then need.
+ These print outputs will then appear in the 'Observation:' field, which will be available as input for the next step.
+ In the end, if a completion tool is available (e.g., finish or a submit_* tool), call it to deliver the final answer. Read that tool's description for the exact required format. You will be generating a JSON object with the following structure:
+ ```json
+ {
+ "thought": "...",
+ "code": "..."
+ }
+ ```
+
+ Here are a few examples using notional tools (note: "finish" in examples is a placeholder for the actual completion tool provided in your tools list; use the real tool name and follow its description):
+ ---
+ Task: "Generate an image of the oldest person in this document."
+
+ {"thought": "I will proceed step by step and use the following tools: `document_qa` to find the oldest person in the document, then `image_generator` to generate an image according to the answer.", "code": "answer = document_qa(document=document, question=\"Who is the oldest person mentioned?\")\nprint(answer)\n"}
+ Observation: "The oldest person in the document is John Doe, a 55 year old lumberjack living in Newfoundland."
+
+ {"thought": "I will now generate an image showcasing the oldest person.", "code": "image = image_generator(\"A portrait of John Doe, a 55-year-old man living in Canada.\")\nfinish(image)\n"}
+ ---
+ Task: "What is the result of the following operation: 5 + 3 + 1294.678?"
+
+ {"thought": "I will use python code to compute the result of the operation and then return the final answer using the completion tool.", "code": "result = 5 + 3 + 1294.678\nfinish(result)\n"}
+
+ ---
+ Task:
+ In a 1979 interview, Stanislaus Ulam discusses with Martin Sherwin about other great physicists of his time, including Oppenheimer.
+ What does he say was the consequence of Einstein learning too much math on his creativity, in one word?
+
+ {"thought": "I need to find and read the 1979 interview of Stanislaus Ulam with Martin Sherwin.", "code": "pages = web_search(query=\"1979 interview Stanislaus Ulam Martin Sherwin physicists Einstein\")\nprint(pages)\n"}
+ Observation:
+ No result found for query "1979 interview Stanislaus Ulam Martin Sherwin physicists Einstein".
+
+ {"thought": "The query was maybe too restrictive and did not find any results. Let's try again with a broader query.", "code": "pages = web_search(query=\"1979 interview Stanislaus Ulam\")\nprint(pages)\n"}
+ Observation:
+ Found 6 pages:
+ [Stanislaus Ulam 1979 interview](https://ahf.nuclearmuseum.org/voices/oral-histories/stanislaus-ulams-interview-1979/)
+
+ [Ulam discusses Manhattan Project](https://ahf.nuclearmuseum.org/manhattan-project/ulam-manhattan-project/)
+
+ (truncated)
+
+ {"thought": "I will read the first 2 pages to know more.", "code": "for url in [\"https://ahf.nuclearmuseum.org/voices/oral-histories/stanislaus-ulams-interview-1979/\", \"https://ahf.nuclearmuseum.org/manhattan-project/ulam-manhattan-project/\"]:\n whole_page = visit_webpage(url)\n print(whole_page)\n print(\"\n\" + \"=\"*80 + \"\n\") # Print separator between pages"}
+
+ Observation:
+ Manhattan Project Locations:
+ Los Alamos, NM
+ Stanislaus Ulam was a Polish-American mathematician. He worked on the Manhattan Project at Los Alamos and later helped design the hydrogen bomb. In this interview, he discusses his work at
+ (truncated)
+
+ {"thought": "I now have the final answer: from the webpages visited, Stanislaus Ulam says of Einstein: \"He learned too much mathematics and sort of diminished, it seems to me personally, it seems to me his purely physics creativity.\" Let's answer in one word.", "code": "finish(\"diminished\")"}
+
+ ---
+ Task: "Which city has the highest population: Guangzhou or Shanghai?"
+
+ {"thought": "I need to get the populations for both cities and compare them: I will use the tool `web_search` to get the population of both cities.", "code": "for city in [\"Guangzhou\", \"Shanghai\"]:\n print(f\"Population {city}:\", web_search(f\"{city} population\")"}
+ Observation:
+ Population Guangzhou: ['Guangzhou has a population of 15 million inhabitants as of 2021.']
+ Population Shanghai: '26 million (2019)'
+
+ {"thought": "Now I know that Shanghai has the highest population.", "code": "finish(\"Shanghai\")"}
+
+ ---
+ Task: "What is the current age of the pope, raised to the power 0.36?"
+
+ {"thought": "I will use the tool `wikipedia_search` to get the age of the pope, and confirm that with a web search.", "code": "pope_age_wiki = wikipedia_search(query=\"current pope age\")\nprint(\"Pope age as per wikipedia:\", pope_age_wiki)\npope_age_search = web_search(query=\"current pope age\")\nprint(\"Pope age as per google search:\", pope_age_search)"}
+ Observation:
+ Pope age: "The pope Francis is currently 88 years old."
+
+ {"thought": "I know that the pope is 88 years old. Let's compute the result using python code.", "code": "pope_current_age = 88 ** 0.36\nfinish(pope_current_age)"}
+
+ Above example were using notional tools that might not exist for you. On top of performing computations in the Python code snippets that you create, you only have access to these tools, behaving like regular python functions:
+ ```python
+ {%- for tool in tools.values() %}
+ {{ tool.to_code_prompt() }}
+ {% endfor %}
+ ```
+
+ {%- if managed_agents and managed_agents.values() | list %}
+ You can also give tasks to team members.
+ Calling a team member works similarly to calling a tool: provide the task description as the 'task' argument. Since this team member is a real human, be as detailed and verbose as necessary in your task description.
+ You can also include any relevant variables or context using the 'additional_args' argument.
+ Here is a list of the team members that you can call:
+ ```python
+ {%- for agent in managed_agents.values() %}
+ def {{ agent.name }}(task: str, additional_args: dict[str, Any]) -> str:
+ """{{ agent.description }}
+
+ Args:
+ task: Long detailed description of the task.
+ additional_args: Dictionary of extra inputs to pass to the managed agent, e.g. images, dataframes, or any other contextual data it may need.
+ """
+ {% endfor %}
+ ```
+ {%- endif %}
+
+ {%- if custom_instructions %}
+ {{custom_instructions}}
+ {%- endif %}
+
+ Here are the rules you should always follow to solve your task:
+ 1. Use only variables that you have defined!
+ 2. Always use the right arguments for the tools. DO NOT pass the arguments as a dict as in 'answer = wikipedia_search({'query': "What is the place where James Bond lives?"})', but use the arguments directly as in 'answer = wikipedia_search(query="What is the place where James Bond lives?")'.
+ 3. Take care to not chain too many sequential tool calls in the same code block, especially when the output format is unpredictable. For instance, a call to wikipedia_search has an unpredictable return format, so do not have another tool call that depends on its output in the same block: rather output results with print() to use them in the next block.
+ 4. Call a tool only when needed, and never re-do a tool call that you previously did with the exact same parameters.
+ 5. Don't name any new variable with the same name as a tool: for instance don't name a variable 'final_answer'.
+ 6. Never create any notional variables in our code, as having these in your logs will derail you from the true variables.
+ 7. You can use imports in your code, but only from the following list of modules: {{authorized_imports}}
+ 8. The state persists between code executions: so if in one step you've created variables or imported modules, these will all persist.
+ 9. Don't give up! You're in charge of solving the task, not providing directions to solve it.
+
+ Now Begin!
+planning:
+ initial_plan: |-
+ You are a world expert at analyzing a situation to derive facts, and plan accordingly towards solving a task.
+ Below I will present you a task. You will need to 1. build a survey of facts known or needed to solve the task, then 2. make a plan of action to solve the task.
+
+ ## 1. Facts survey
+ You will build a comprehensive preparatory survey of which facts we have at our disposal and which ones we still need.
+ These "facts" will typically be specific names, dates, values, etc. Your answer should use the below headings:
+ ### 1.1. Facts given in the task
+ List here the specific facts given in the task that could help you (there might be nothing here).
+
+ ### 1.2. Facts to look up
+ List here any facts that we may need to look up.
+ Also list where to find each of these, for instance a website, a file... - maybe the task contains some sources that you should reuse here.
+
+ ### 1.3. Facts to derive
+ List here anything that we want to derive from the above by logical reasoning, for instance computation or simulation.
+
+ Don't make any assumptions. For each item, provide a thorough reasoning. Do not add anything else on top of three headings above.
+
+ ## 2. Plan
+ Then for the given task, develop a step-by-step high-level plan taking into account the above inputs and list of facts.
+ This plan should involve individual tasks based on the available tools, that if executed correctly will yield the correct answer.
+ Do not skip steps, do not add any superfluous steps. Only write the high-level plan, DO NOT DETAIL INDIVIDUAL TOOL CALLS.
+ After writing the final step of the plan, write the '' tag and stop there.
+
+ You can leverage these tools, behaving like regular python functions:
+ ```python
+ {%- for tool in tools.values() %}
+ {{ tool.to_code_prompt() }}
+ {% endfor %}
+ ```
+
+ {%- if managed_agents and managed_agents.values() | list %}
+ You can also give tasks to team members.
+ Calling a team member works similarly to calling a tool: provide the task description as the 'task' argument. Since this team member is a real human, be as detailed and verbose as necessary in your task description.
+ You can also include any relevant variables or context using the 'additional_args' argument.
+ Here is a list of the team members that you can call:
+ ```python
+ {%- for agent in managed_agents.values() %}
+ def {{ agent.name }}(task: str, additional_args: dict[str, Any]) -> str:
+ """{{ agent.description }}
+
+ Args:
+ task: Long detailed description of the task.
+ additional_args: Dictionary of extra inputs to pass to the managed agent, e.g. images, dataframes, or any other contextual data it may need.
+ """
+ {% endfor %}
+ ```
+ {%- endif %}
+
+ ---
+ Now begin! Here is your task:
+ ```
+ {{task}}
+ ```
+ First in part 1, write the facts survey, then in part 2, write your plan.
+ update_plan_pre_messages: |-
+ You are a world expert at analyzing a situation, and plan accordingly towards solving a task.
+ You have been given the following task:
+ ```
+ {{task}}
+ ```
+
+ Below you will find a history of attempts made to solve this task.
+ You will first have to produce a survey of known and unknown facts, then propose a step-by-step high-level plan to solve the task.
+ If the previous tries so far have met some success, your updated plan can build on these results.
+ If you are stalled, you can make a completely new plan starting from scratch.
+
+ Find the task and history below:
+ update_plan_post_messages: |-
+ Now write your updated facts below, taking into account the above history:
+ ## 1. Updated facts survey
+ ### 1.1. Facts given in the task
+ ### 1.2. Facts that we have learned
+ ### 1.3. Facts still to look up
+ ### 1.4. Facts still to derive
+
+ Then write a step-by-step high-level plan to solve the task above.
+ ## 2. Plan
+ ### 2. 1. ...
+ Etc.
+ This plan should involve individual tasks based on the available tools, that if executed correctly will yield the correct answer.
+ Beware that you have {remaining_steps} steps remaining.
+ Do not skip steps, do not add any superfluous steps. Only write the high-level plan, DO NOT DETAIL INDIVIDUAL TOOL CALLS.
+ After writing the final step of the plan, write the '' tag and stop there.
+
+ You can leverage these tools, behaving like regular python functions:
+ ```python
+ {%- for tool in tools.values() %}
+ {{ tool.to_code_prompt() }}
+ {% endfor %}
+ ```
+
+ {%- if managed_agents and managed_agents.values() | list %}
+ You can also give tasks to team members.
+ Calling a team member works similarly to calling a tool: provide the task description as the 'task' argument. Since this team member is a real human, be as detailed and verbose as necessary in your task description.
+ You can also include any relevant variables or context using the 'additional_args' argument.
+ Here is a list of the team members that you can call:
+ ```python
+ {%- for agent in managed_agents.values() %}
+ def {{ agent.name }}(task: str, additional_args: dict[str, Any]) -> str:
+ """{{ agent.description }}
+
+ Args:
+ task: Long detailed description of the task.
+ additional_args: Dictionary of extra inputs to pass to the managed agent, e.g. images, dataframes, or any other contextual data it may need.
+ """
+ {% endfor %}
+ ```
+ {%- endif %}
+
+ Now write your updated facts survey below, then your new plan.
+managed_agent:
+ task: |-
+ You're a helpful agent named '{{name}}'.
+ You have been submitted this task by your manager.
+ ---
+ Task:
+ {{task}}
+ ---
+ You're helping your manager solve a wider task: so make sure to not provide a one-line answer, but give as much information as possible to give them a clear understanding of the answer.
+
+ Your final submission via the completion tool MUST contain these parts unless the tool description specifies a different format (in which case, follow the tool description exactly):
+ ### 1. Task outcome (short version):
+ ### 2. Task outcome (extremely detailed version):
+ ### 3. Additional context (if relevant):
+
+ Put all these in the completion tool call, and read the tool's description for the required argument names and format. Everything that you do not pass as an argument to that tool will be lost.
+ And even if your task resolution is not successful, please return as much context as possible, so that your manager can act upon this feedback.
+ report: |-
+ Here is the final answer from your managed agent '{{name}}':
+ {{final_answer}}
+final_answer:
+ pre_messages: |-
+ An agent tried to answer a user query but it got stuck and failed to do so. You are tasked with providing an answer instead. Here is the agent's memory:
+ post_messages: |-
+ Based on the above, please provide an answer to the following user task:
+ {{task}}
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/tool_calling_agent.py b/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/tool_calling_agent.py
new file mode 100644
index 00000000..7f3bccb7
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/tool_calling_agent.py
@@ -0,0 +1,21 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from typing import ClassVar
+
+from .base_agent import SmolagentBaseAgent
+
+
+class SmolagentToolCallingAgent(SmolagentBaseAgent):
+ display_name: ClassVar[str] = "SmolAgents Tool Calling"
+ slug_name: ClassVar[str] = "smolagents_tool"
+
+ @classmethod
+ def _get_instance_class(cls):
+ from .tool_calling_instance import SmolagentToolCallingAgentInstance
+
+ return SmolagentToolCallingAgentInstance
+
+ @classmethod
+ def _get_instance_class_ref(cls) -> str:
+ return "exgentic.agents.smolagents.tool_calling_instance:SmolagentToolCallingAgentInstance"
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/tool_calling_instance.py b/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/tool_calling_instance.py
new file mode 100644
index 00000000..9426b003
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/smolagents/tool_calling_instance.py
@@ -0,0 +1,37 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from smolagents import ToolCallingAgent
+from smolagents.tools import Tool
+from smolagents.utils import AgentError
+
+from .base_instance import SmolagentBaseAgentInstance
+
+
+class SmolagentToolCallingAgentInstance(SmolagentBaseAgentInstance):
+ """Smolagent implementation."""
+
+ def run_smolagent(self, tools: list[Tool]):
+ self._agent = ToolCallingAgent(
+ tools=tools,
+ model=self.get_internal_model(),
+ # use_structured_outputs_internally=True,
+ logger=self.get_smolagent_logger(),
+ )
+
+ prompt = f"Task: {self.task}\n\n"
+ if self.context:
+ prompt += f"Context: {self.context}\n\n"
+ prompt += (
+ "Complete this task using the available tools. "
+ "Each tool corresponds to an action you can take in the environment.\n"
+ )
+ if self.initial_observation is not None and not self.initial_observation.is_empty():
+ text = str(self.initial_observation).strip()
+ if text:
+ prompt += f"\nFirst Observation: {text}\n"
+ try:
+ self._agent.run(task=prompt, max_steps=self.max_steps)
+ except AgentError as e:
+ self.logger.info(f"AgentError: {e}")
+ raise
diff --git a/labs/AgentStream/exgentic/src/exgentic/agents/tool_shortlisting.py b/labs/AgentStream/exgentic/src/exgentic/agents/tool_shortlisting.py
new file mode 100644
index 00000000..25ee84de
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/agents/tool_shortlisting.py
@@ -0,0 +1,101 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The AgentStream organization and its contributors.
+
+from __future__ import annotations
+
+import logging
+from typing import Any, Callable, List
+
+from litellm import (
+ ChatCompletionDeveloperMessage,
+ ChatCompletionUserMessage,
+)
+
+
+def shortlist_tools(
+ tools: list[dict[str, Any]],
+ max_selected: int,
+ messages: list[Any],
+ completion_fn: Callable[..., Any],
+ model: str,
+ logger: logging.Logger,
+ *,
+ cost_callback: Callable[[Any], None] | None = None,
+) -> list[dict[str, Any]]:
+ if len(tools) <= max_selected:
+ return tools
+
+ logger.info("Tool shortlisting: %d available -> selecting top %d", len(tools), max_selected)
+
+ names = [tool["function"]["name"] for tool in tools]
+ names_str = ""
+ for tool in tools:
+ names_str += f"\n- {tool['function']['name']}: {tool['function']['description']}"
+
+ history_text = _render_history(messages)
+
+ dev = ChatCompletionDeveloperMessage(
+ role="developer",
+ content=(
+ f"Please before providing your next move list the names of the top "
+ f"{max_selected} tools that are somewhat relevant for the next step, "
+ "ordered by relevancy (most to least). Return ONLY a JSON object with this shape: "
+ '{\n "tools": ["tool_name_1", "tool_name_2", ...]\n}.\n'
+ f"Choose from these tools only: {names_str}.\n"
+ f"Do not call any of those tools just return the list of the top "
+ f"{max_selected} relevant tools names in the required format."
+ ),
+ )
+ history_msg = ChatCompletionUserMessage(
+ role="user",
+ content=f"Conversation so far (plain text):\n{history_text}",
+ )
+
+ try:
+ response = completion_fn(model=model, messages=[dev, history_msg])
+ except Exception as exc:
+ logger.warning("Tool shortlisting LLM call failed: %s", exc)
+ return tools[:max_selected]
+
+ if cost_callback and response and response.usage:
+ cost_callback(response.usage)
+
+ text = response.choices[0].message.content
+ if text is None:
+ text = str(response.choices[0].message)
+
+ positions = []
+ for name in names:
+ idx = text.find(name)
+ if idx != -1:
+ positions.append((idx, name))
+
+ if len(positions) == 0:
+ logger.info("Tool shortlist fallback: no matches, taking first %d", max_selected)
+ return tools[:max_selected]
+
+ positions.sort(key=lambda x: x[0])
+ selected_names = [name for _, name in positions][:max_selected]
+ name_to_tool = {tool["function"]["name"]: tool for tool in tools}
+ selected_tools = [name_to_tool[name] for name in selected_names]
+ logger.info("Tool shortlist: %d -> %d", len(tools), len(selected_tools))
+ return selected_tools
+
+
+def _render_history(messages: list[Any]) -> str:
+ parts: List[str] = []
+ for message in messages:
+ msg = message if isinstance(message, dict) else dict(message)
+ role = msg.get("role") or "unknown"
+ if role == "tool":
+ content = msg.get("content", "")
+ parts.append(f"tool: {content}")
+ continue
+ content = msg.get("content")
+ if content:
+ parts.append(f"{role}: {content}")
+ tool_calls = msg.get("tool_calls") or []
+ for tc in tool_calls:
+ fn = tc.get("function", {}) if isinstance(tc, dict) else {}
+ parts.append(f"{role} tool_call: {fn.get('name', '?')}({fn.get('arguments', '')})")
+ return "\n".join(parts)[-8000:]
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/__init__.py b/labs/AgentStream/exgentic/src/exgentic/benchmarks/__init__.py
new file mode 100644
index 00000000..367fec41
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/__init__.py
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/appworld/__init__.py b/labs/AgentStream/exgentic/src/exgentic/benchmarks/appworld/__init__.py
new file mode 100644
index 00000000..4fa31ba8
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/appworld/__init__.py
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from .appworld_benchmark import AppWorldBenchmark
+
+__all__ = [
+ "AppWorldBenchmark",
+]
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/appworld/appworld_benchmark.py b/labs/AgentStream/exgentic/src/exgentic/benchmarks/appworld/appworld_benchmark.py
new file mode 100644
index 00000000..a7d5c95a
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/appworld/appworld_benchmark.py
@@ -0,0 +1,57 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+"""AppWorld benchmark adapter -- light benchmark class only.
+
+Evaluator and session classes live in ``appworld_eval.py`` and are loaded
+inside the runner subprocess via ``_get_evaluator_class()`` and
+``_get_session_class()``. This file must remain importable without the
+``appworld`` package installed.
+"""
+
+from __future__ import annotations
+
+from typing import Any, ClassVar, Literal
+
+from pydantic import BaseModel, ConfigDict, Field
+
+from ...core.benchmark import Benchmark
+from ...core.types import SingleObservation
+
+
+class AppWorldObservation(SingleObservation):
+ pass
+
+
+class AppWorldBenchmark(Benchmark, BaseModel):
+ display_name: ClassVar[str] = "AppWorld"
+ slug_name: ClassVar[str] = "appworld"
+ available_subsets: ClassVar[list[str]] = ["train", "dev", "test_normal", "test_challenge"]
+ model_config = ConfigDict(arbitrary_types_allowed=True)
+
+ @classmethod
+ def _get_evaluator_class(cls):
+ return "exgentic.benchmarks.appworld.appworld_eval:AppWorldEvaluator"
+
+ @classmethod
+ def _get_session_class(cls):
+ return "exgentic.benchmarks.appworld.appworld_eval:AppWorldSession"
+
+ # Inputs
+ subset: Literal["train", "dev", "test_normal", "test_challenge"] = "test_challenge"
+ env_kwargs: dict[str, Any] = Field(default_factory=dict)
+ max_interactions: int = 200
+ tool_name_separator: Literal[".", "__"] = "__"
+ SCORES_FILE_NAME: ClassVar[str] = "scores.json"
+
+ def list_subsets(self) -> list[str]: # type: ignore[override]
+ return list(self.available_subsets)
+
+ def _get_evaluator_kwargs(self) -> dict[str, Any]:
+ return {
+ "subset": self.subset,
+ "env_kwargs": self.env_kwargs,
+ "max_interactions": self.max_interactions,
+ "tool_name_separator": self.tool_name_separator,
+ "use_cache": self.use_cache,
+ }
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/appworld/appworld_eval.py b/labs/AgentStream/exgentic/src/exgentic/benchmarks/appworld/appworld_eval.py
new file mode 100644
index 00000000..535abcd3
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/appworld/appworld_eval.py
@@ -0,0 +1,673 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+"""AppWorld evaluator and session classes.
+
+These classes import the ``appworld`` package at method level. They are only
+ever instantiated inside the isolated runner subprocess, so the heavy
+dependency is never required in the host process.
+
+The light ``AppWorldBenchmark`` class lives in ``appworld_benchmark.py`` and
+must remain importable without the ``appworld`` package installed.
+"""
+
+from __future__ import annotations
+
+import json
+import logging
+import shutil
+from pathlib import Path
+from shutil import copytree
+from typing import TYPE_CHECKING, Any, ClassVar, Literal
+
+from pydantic import (
+ BaseModel,
+ create_model,
+)
+
+from ...core.actions import ActionsHandler
+from ...core.evaluator import Evaluator
+from ...core.session import Session
+from ...core.types import (
+ Action,
+ ActionType,
+ BenchmarkResults,
+ EmptyObservation,
+ FinishAction,
+ MessageAction,
+ Observation,
+ SessionIndex,
+ SessionScore,
+ SingleAction,
+)
+from ...utils.paths import get_run_id, get_run_paths
+from ...utils.settings import get_settings
+from .appworld_benchmark import AppWorldObservation
+
+settings = get_settings()
+logger = logging.getLogger(__name__)
+
+APPWORLD_TOTAL_TASKS = {
+ "train": 90,
+ "dev": 57,
+ "test_normal": 168,
+ "test_challenge": 417,
+}
+
+if TYPE_CHECKING:
+ from appworld.environment import AppWorld # type: ignore
+
+
+class AppWorldSession(Session):
+ """Session that hosts AppWorld directly (no separate WorldProcess).
+
+ Tools are derived from the AppWorld task API docs and mapped to ActionTypes.
+ Each step calls AppWorld.requester.request(app, api, **args) and wraps the
+ result as observations, surfacing API errors as structured payloads.
+ """
+
+ CACHE_DIR: ClassVar[str] = "./appworld_disk_cache"
+ TASK_OUTPUT_SUBDIR: ClassVar[str] = "task_output"
+ SCORES_FILE_NAME: ClassVar[str] = "scores.json"
+
+ def __init__(
+ self,
+ session_id: str | None = None,
+ task_spec: dict[str, Any] | None = None,
+ env_kwargs: dict[str, Any] | None = None,
+ use_cache: bool = True,
+ tool_name_separator: str = ".",
+ max_interactions: int | None = None,
+ ) -> None:
+ if session_id is not None:
+ self._session_id = session_id
+ self._task_spec = task_spec or {}
+ self._env_kwargs = env_kwargs or {}
+ self._tool_name_separator = tool_name_separator
+ self._max_interactions = max_interactions
+ self._action_count = 0
+ self._registry = ActionsHandler(
+ logger=self.logger,
+ warn_on_validation_error=False,
+ warn_on_unknown_action=False,
+ handle_validation_error=lambda action, _msg: self.apply_action(action),
+ handle_unknown_action=self.apply_action,
+ )
+ self._step_count: int = 0
+ self._done: bool = False
+ self._world_closed: bool = False
+ self._cached_score: SessionScore | None = None
+
+ # Resolve task_id
+ task_id = self._task_spec.get("task_id") if isinstance(self._task_spec, dict) else None
+ if isinstance(self._task_spec, str):
+ task_id = self._task_spec
+ if not task_id:
+ raise ValueError("AppWorldSession requires task_spec with 'task_id' or be a task_id string")
+ self._task_id = str(task_id)
+
+ # Construct AppWorld in-process (lazy import to defer side effects)
+ from appworld import update_root # type: ignore
+ from appworld.common.constants import DEFAULT_EXPERIMENT_NAME # type: ignore
+ from appworld.common.path_store import path_store # type: ignore
+ from appworld.environment import AppWorld # type: ignore
+
+ # Point appworld at the correct data directory before loading the task.
+ cache = Path(settings.cache_dir).expanduser()
+ update_root(str(cache / "appworld"))
+
+ # Patch appworld's SQLite connection helper to allow cross-thread usage.
+ # The venv runner serves via uvicorn which may dispatch requests across
+ # threads, but appworld's @lru_cache'd connections default to
+ # check_same_thread=True, causing ProgrammingError.
+ self._patch_appworld_sqlite()
+
+ self._world: AppWorld = AppWorld(task_id=self._task_id, **self._env_kwargs)
+ self._experiment_name: str = self._world.experiment_name or DEFAULT_EXPERIMENT_NAME
+ self._task_output_dir: Path = (
+ Path(path_store.experiment_outputs) / self._experiment_name / "tasks" / self._task_id
+ )
+
+ self.logger.info(f"Task ID: {task_spec}")
+ super().__init__()
+
+ @staticmethod
+ def _patch_appworld_sqlite() -> None:
+ """Patch appworld's SQLite helpers to tolerate cross-thread access.
+
+ appworld's ``get_direct_sqlite3_connection`` creates connections with
+ the default ``check_same_thread=True``, then caches them via
+ ``@lru_cache``. When the uvicorn runner dispatches requests on
+ different threads, reusing those connections raises
+ ``sqlite3.ProgrammingError``. We patch the function and clear the
+ cache so fresh connections are created with ``check_same_thread=False``.
+ """
+ import sqlite3 as _sqlite3
+
+ from appworld.apps.lib.models import db as _appworld_db # type: ignore
+
+ _original = _appworld_db.get_direct_sqlite3_connection
+
+ if getattr(_original, "_exgentic_patched", False):
+ return
+
+ def _safe_connect(db_app_path: str) -> _sqlite3.Connection:
+ conn = _sqlite3.connect(db_app_path, check_same_thread=False)
+ conn.execute("PRAGMA mmap_size = 268435456")
+ return conn
+
+ _safe_connect._exgentic_patched = True # type: ignore[attr-defined]
+ _appworld_db.get_direct_sqlite3_connection = _safe_connect
+ # Clear the lru_cache so stale thread-bound connections aren't reused.
+ _appworld_db.get_direct_cached_sqlite3_connection.cache_clear()
+
+ def get_config(self) -> dict[str, Any]:
+ return {
+ "task_spec": self._task_spec,
+ "env_kwargs": self._env_kwargs,
+ "tool_name_separator": self._tool_name_separator,
+ "max_interactions": self._max_interactions,
+ }
+
+ @property
+ def world(self) -> AppWorld:
+ return self._world
+
+ @property
+ def task(self) -> str:
+ self.logger.info(f"Task: {self.world.task.instruction}")
+ return "Task from supervisor:\n" + self.world.task.instruction
+
+ @property
+ def context(self) -> dict[str, Any]:
+ if self.world.task is None:
+ raise ValueError("AppWorld task is not initialized")
+
+ allowed = ", ".join(app for app in self.world.task.allowed_apps if app != "api_docs")
+ return {
+ "policy": (
+ "This environment provides a set of applications,"
+ " each exposing a predefined set of APIs that may"
+ " be used to perform tasks on behalf of the"
+ " supervisor. The applications include:"
+ f" {allowed}.\n"
+ " The available applications and their APIs are"
+ " fixed for the task.\n"
+ "\n"
+ "Supervisor account credentials (such as emails,"
+ " usernames, and passwords) are available through"
+ " the supervisor application's APIs and are"
+ " accessed from there when required.\n"
+ "\n"
+ "If an application requires an access token to"
+ " perform authenticated operations, the access"
+ " token is obtained by calling that application's"
+ " authentication/login API using the credentials"
+ " retrieved from the supervisor application."
+ " Access tokens are not provided by the supervisor"
+ " application.\n"
+ "\n"
+ "References to people (e.g., friends, family,"
+ " roommates) correspond to entries in the"
+ " phone_contacts application.\n"
+ "References to files or storage correspond to the"
+ " file_system application, not the local machine"
+ " filesystem.\n"
+ "\n"
+ "Time-based instructions (e.g., 'this month',"
+ " 'yesterday') are interpreted with full calendar"
+ " boundary ranges.\n"
+ "If an API returns paginated results, all pages"
+ " constitute the complete result.\n"
+ "\n"
+ "The environment consists only of the provided"
+ " applications and their documented APIs and"
+ " parameters. No additional endpoints, methods,"
+ " arguments, or capabilities are assumed beyond"
+ " those explicitly defined.\n"
+ "\n"
+ "When task execution is finished, the designated"
+ " task-completion API is used to signal completion."
+ " If the task requires a final answer value, the"
+ " answer is returned through that completion API."
+ " If the task cannot be completed using the"
+ " available applications and APIs, the task may be"
+ " marked as failed."
+ ),
+ "supervisor": dict(self.world.task.supervisor),
+ # "app_descriptions": self.world.task.app_descriptions,
+ # "allowed_apps": self.world.task.allowed_apps,
+ "datetime": self.world.task.datetime.isoformat(),
+ }
+
+ @property
+ def actions(self) -> list[ActionType]:
+ if not self._registry.actions:
+ # Build ActionTypes from AppWorld function_calling docs to leverage enriched auth parameters
+ from ...adapters.schemas.json_schema import make_args_model_from_json_schema
+
+ tools_specs = self.world.task.api_docs.function_calling()
+ for tool in tools_specs:
+ function = tool["function"]
+ raw_name = function["name"]
+ separator = self._tool_name_separator
+ name = raw_name.replace("__", separator)
+ app, api = name.split(separator, 1)
+ if app == "api_docs":
+ continue
+ if api == "show_active_task":
+ continue
+
+ args_model = make_args_model_from_json_schema(name, function["parameters"])
+
+ if raw_name == "supervisor__complete_task":
+ finish_act = create_model(
+ "AppWorldFinishAction",
+ __base__=FinishAction,
+ arguments=(args_model, ...),
+ )
+ self._registry.add_action(
+ name="finish",
+ description=function["description"],
+ action_cls=finish_act,
+ handler=self.apply_action,
+ is_finish=True,
+ )
+ else:
+ act = create_model(
+ f"{name}_Action",
+ __base__=SingleAction,
+ name=(Literal[name], name),
+ arguments=(args_model, ...),
+ )
+ self._registry.add_action(
+ name=name,
+ description=function["description"],
+ action_cls=act,
+ handler=self.apply_action,
+ )
+ return self._registry.actions
+
+ @property
+ def task_id(self) -> str:
+ return str(self._task_id)
+
+ @property
+ def _actions_names(self) -> set[str]:
+ return {a.name for a in self.actions}
+
+ def _to_observation(self, raw: Any, invoking: list[SingleAction] | None = None) -> Observation:
+ return AppWorldObservation(invoking_actions=invoking or [], result=raw)
+
+ def start(self) -> Observation | None:
+ self.logger.info(f"session_start id={self.session_id} task_id={self._task_id}")
+ # Empty initial observation; task details are provided via task/context.
+ return EmptyObservation()
+
+ def _is_message_action(self, action: SingleAction) -> bool:
+ if isinstance(action, MessageAction) or action.name == "message":
+ return True
+ return False
+
+ def apply_action(self, action: SingleAction):
+ if self._is_message_action(action):
+ self._step_count += 1
+ return AppWorldObservation(
+ invoking_actions=[action],
+ result="Error: Sending a message is not allowed. Please use only one of the available actions.",
+ )
+ # if action.name not in self._actions_names:
+ # return AppWorldObservation(invoking_actions=[action], result="Wrong name: {action.name}")
+ # Map benchmark-level finish to the supervisor.complete_task endpoint
+ effective_name = action.name
+ if action.name == "finish":
+ separator = self._tool_name_separator
+ effective_name = f"supervisor{separator}complete_task"
+
+ separator = self._tool_name_separator
+ parts = effective_name.split(separator, 1)
+ if len(parts) != 2:
+ parts = effective_name, ""
+ app_name, api_name = parts
+
+ arguments = action.arguments
+ if isinstance(arguments, BaseModel):
+ arguments = arguments.model_dump()
+
+ self.logger.info(f"App: {app_name}, Function: {api_name}, Arguments: {arguments}")
+
+ try:
+ out = self.world.requester.request(app_name, api_name, **arguments)
+ except Exception as e:
+ try:
+ e = json.loads(str(e).split("\n")[-1])["message"]
+ except json.JSONDecodeError:
+ pass
+ out = "Error: " + str(e)
+ finally:
+ self._step_count += 1
+
+ self.logger.info(f"Output: {out}")
+
+ return AppWorldObservation(
+ invoking_actions=[action],
+ result=out,
+ )
+
+ @staticmethod
+ def _max_interactions_error(observation: Observation) -> bool:
+ for obs in observation.to_observation_list():
+ result = obs.result
+ if isinstance(result, str) and "Maximum number of executions" in result:
+ return True
+ return False
+
+ def step(self, action: Action) -> Observation | None:
+ if self._done:
+ return None
+
+ if self._max_interactions is not None:
+ incoming = len(action.to_action_list())
+ if self._action_count + incoming > self._max_interactions:
+ self.logger.warning(
+ "AppWorld local max_interactions reached (%s/%s); terminating session",
+ self._action_count,
+ self._max_interactions,
+ )
+ return None
+
+ observation = self._registry.execute(action)
+ if observation is None:
+ return None
+ if self._max_interactions is not None:
+ self._action_count += len(action.to_action_list())
+ if self._max_interactions_error(observation):
+ self.logger.warning(
+ "AppWorld max_interactions reached (%s/%s); terminating session",
+ self.world.num_interactions,
+ self.world.max_interactions,
+ )
+ return None
+ return observation
+
+ def done(self) -> bool:
+ return self.world.task_completed()
+
+ def score(self) -> SessionScore:
+ if self._cached_score is not None:
+ return self._cached_score
+ # World was already saved and closed in close(); compute the actual evaluation score now.
+ from appworld.apps.lib.models.db import CachedDBHandler
+ from appworld.evaluator import evaluate_task
+
+ test_tracker = evaluate_task(
+ task_id=self._task_id,
+ experiment_name=self._experiment_name,
+ suppress_errors=True,
+ save_report=False,
+ )
+ score_value = float(test_tracker.pass_percentage) / 100.0
+ self.logger.info(
+ "Evaluation results: pass_percentage=%s pass_count=%s fail_count=%s num_tests=%s success=%s",
+ test_tracker.pass_percentage,
+ test_tracker.pass_count,
+ test_tracker.fail_count,
+ test_tracker.num_tests,
+ test_tracker.success,
+ )
+
+ # Check task completion before evaluate_task potentially closes DB.
+ try:
+ finished = self.world.task_completed()
+ except Exception:
+ finished = bool(self._done)
+ # Reset cached DB handler for this task so later aggregate evaluation can run.
+ CachedDBHandler.reset(self._task_id)
+ # Surface benchmark evaluation details for downstream analysis.
+ session_metrics = {
+ "pass_percentage": test_tracker.pass_percentage,
+ "pass_count": test_tracker.pass_count,
+ "fail_count": test_tracker.fail_count,
+ "num_tests": test_tracker.num_tests,
+ "difficulty": test_tracker.difficulty,
+ "success": test_tracker.success,
+ }
+ tracker_dict = test_tracker.to_dict(stats_only=False)
+ scores_path = self.paths.benchmark_dir / self.SCORES_FILE_NAME
+ scores_path.parent.mkdir(parents=True, exist_ok=True)
+ with open(scores_path, "w", encoding="utf-8") as f:
+ json.dump(
+ {
+ "task_id": self._task_id,
+ "session_id": self.session_id,
+ "test_tracker": tracker_dict,
+ **session_metrics,
+ },
+ f,
+ ensure_ascii=False,
+ indent=2,
+ )
+ session_metadata = {"test_tracker": tracker_dict}
+ sc = SessionScore(
+ score=score_value,
+ success=test_tracker.success,
+ is_finished=finished,
+ session_metrics=session_metrics,
+ session_metadata=session_metadata,
+ )
+ # Cache here so a later close()->score() call does not re-run
+ # evaluate_task() (which fails with IndexError after the task DB has
+ # already been reset by the first evaluation).
+ self._cached_score = sc
+ return sc
+
+ def close(self):
+ # Save AppWorld task state and mirror logs
+ self.logger.info(
+ "Closing AppWorld session: steps=%s done=%s world_closed=%s",
+ self._step_count,
+ self._done,
+ self._world_closed,
+ )
+ if self._world_closed:
+ self.logger.warning("AppWorld session close called more than once.")
+ try:
+ self.world.save()
+ except Exception:
+ self.logger.exception("AppWorld world.save failed")
+ raise
+ try:
+ self._done = self.world.task_completed()
+ except Exception:
+ # DB may already be closed by score()/evaluate_task; safe to skip.
+ self.logger.debug("AppWorld task_completed check skipped in close (DB likely closed)")
+ logs_src = self._task_output_dir / "logs"
+ if logs_src.exists():
+ dest = self.paths.benchmark_dir / "logs"
+ dest.parent.mkdir(parents=True, exist_ok=True)
+ try:
+ copytree(logs_src, dest, dirs_exist_ok=True)
+ except Exception:
+ self.logger.exception("AppWorld log copy failed")
+ raise
+ task_output_src = self._task_output_dir
+ if task_output_src.exists():
+ task_output_dest = self.paths.benchmark_dir / self.TASK_OUTPUT_SUBDIR
+ try:
+ if task_output_dest.exists():
+ shutil.rmtree(task_output_dest)
+ copytree(task_output_src, task_output_dest)
+ except Exception:
+ self.logger.exception("AppWorld task output copy failed")
+ raise
+ # Write a standardized results.json plus AppWorld-specific fields
+ try:
+ sc = self.score()
+ except Exception:
+ self.logger.exception("AppWorld evaluation failed")
+ raise
+ self._cached_score = sc
+ try:
+ self.save_results(
+ {
+ "score": sc.score,
+ "success": sc.success,
+ "session_id": self.session_id,
+ "task_id": self._task_id,
+ "completed": self._done,
+ "steps": self._step_count,
+ }
+ )
+ except Exception:
+ self.logger.exception("AppWorld save_results failed")
+ raise
+ self.logger.info(
+ "Session Finished | Success: %s, steps=%s, score=%s",
+ self._done,
+ self._step_count,
+ sc.score,
+ )
+ try:
+ self.world.close()
+ except Exception:
+ self.logger.exception("AppWorld world.close failed")
+ raise
+ self._world_closed = True
+ experiment_root = self._task_output_dir.parent.parent
+ if experiment_root.name == self._experiment_name:
+ try:
+ shutil.rmtree(experiment_root)
+ except Exception:
+ self.logger.exception("AppWorld temp experiment cleanup failed")
+
+
+class AppWorldEvaluator(Evaluator):
+ """Evaluator for AppWorld -- task discovery, session config, and aggregation."""
+
+ def __init__(
+ self,
+ subset: str = "test_normal",
+ env_kwargs: dict[str, Any] | None = None,
+ max_interactions: int = 200,
+ tool_name_separator: str = "__",
+ use_cache: bool = True,
+ ) -> None:
+ self._subset = subset
+ self._env_kwargs = env_kwargs or {}
+ self._max_interactions = max_interactions
+ self._tool_name_separator = tool_name_separator
+ self._use_cache = use_cache
+ self._experiment_name: str = ""
+
+ def _ensure_appworld_root(self) -> None:
+ from appworld import update_root # type: ignore
+
+ cache = Path(settings.cache_dir).expanduser()
+ root = str(cache / "appworld")
+ update_root(root)
+
+ def list_tasks(self) -> list[str]:
+ from appworld.task import load_task_ids # type: ignore
+
+ self._ensure_appworld_root()
+ items: list[str] | None = load_task_ids(self._subset)
+ if not items:
+ return []
+ return [str(t) for t in items]
+
+ def get_session_kwargs(self, index: SessionIndex) -> dict[str, Any]:
+ self._ensure_appworld_root()
+ if not self._experiment_name:
+ self._experiment_name = get_run_id()
+ task_id = index.task_id
+ session_id = index.session_id
+ experiment_name = f"{self._experiment_name}__{session_id}"
+ spec = {"task_id": task_id}
+
+ return {
+ "session_id": session_id,
+ "task_spec": spec,
+ "env_kwargs": {
+ **self._env_kwargs,
+ "max_interactions": self._max_interactions,
+ "experiment_name": experiment_name,
+ },
+ "use_cache": self._use_cache,
+ "tool_name_separator": self._tool_name_separator,
+ "max_interactions": self._max_interactions,
+ }
+
+ def _stage_task_outputs(
+ self,
+ *,
+ task_ids: list[str],
+ task_to_session: dict[str, str],
+ temp_output_dir: Path,
+ ) -> None:
+ run_paths = get_run_paths()
+ for task_id in task_ids:
+ session_id = task_to_session.get(task_id)
+ if not session_id:
+ raise FileNotFoundError(f"Missing session mapping for AppWorld task '{task_id}'.")
+ session_task_output = run_paths.session(session_id).benchmark_dir / AppWorldSession.TASK_OUTPUT_SUBDIR
+ if not session_task_output.exists():
+ raise FileNotFoundError(
+ f"Missing staged task output for task='{task_id}' session='{session_id}' at {session_task_output}"
+ )
+ dest_task_dir = temp_output_dir / "tasks" / task_id
+ dest_task_dir.parent.mkdir(parents=True, exist_ok=True)
+ copytree(session_task_output, dest_task_dir)
+
+ def aggregate_sessions(self, sessions: list[SessionIndex]) -> BenchmarkResults:
+ from appworld.evaluator import Metric, TestTracker # type: ignore
+
+ self._ensure_appworld_root()
+
+ if not sessions:
+ return BenchmarkResults(
+ benchmark_name="appworld",
+ total_tasks=0,
+ score=0.0,
+ metrics={},
+ )
+
+ run_paths = get_run_paths()
+ task_id_to_test_tracker: dict[str, TestTracker] = {}
+ for session in sessions:
+ task_id = str(session.task_id)
+ results_path = run_paths.session(session.session_id).results
+ scores_path = run_paths.session(session.session_id).benchmark_dir / AppWorldSession.SCORES_FILE_NAME
+
+ if scores_path.exists():
+ with open(scores_path, encoding="utf-8") as f:
+ scores_payload = json.load(f)
+ tracker = scores_payload.get("test_tracker")
+ else:
+ logger.warning(
+ "Missing AppWorld scores file for task_id=%s session_id=%s at %s; "
+ "falling back to session metadata in %s",
+ task_id,
+ session.session_id,
+ scores_path,
+ results_path,
+ )
+ with open(results_path, encoding="utf-8") as f:
+ payload = json.load(f)
+ tracker = (payload.get("details") or {}).get("session_metadata", {}).get("test_tracker")
+ if not isinstance(tracker, dict):
+ raise ValueError(
+ "Missing test_tracker in aggregation source for "
+ f"task_id={task_id} session_id={session.session_id}. "
+ f"Checked {scores_path} and fallback {results_path}."
+ )
+
+ task_id_to_test_tracker[task_id] = TestTracker.from_dict(tracker, suppress_errors=False)
+
+ evaluation_dict = Metric.compute_metrics(task_id_to_test_tracker, include_details=True)
+ report = Metric.build_report(evaluation_dict)
+ return BenchmarkResults(
+ benchmark_name="appworld",
+ total_tasks=len(sessions),
+ score=evaluation_dict["aggregate"]["task_goal_completion"] / 100,
+ metrics=report,
+ )
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/appworld/requirements.txt b/labs/AgentStream/exgentic/src/exgentic/benchmarks/appworld/requirements.txt
new file mode 100644
index 00000000..3ec94d1c
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/appworld/requirements.txt
@@ -0,0 +1 @@
+appworld @ git+https://github.com/StonyBrookNLP/appworld.git@edc960129fa6889c2b381715ecd108982029f6d1
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/appworld/setup.sh b/labs/AgentStream/exgentic/src/exgentic/benchmarks/appworld/setup.sh
new file mode 100644
index 00000000..cf9575ba
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/appworld/setup.sh
@@ -0,0 +1,25 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+if ! command -v git-lfs >/dev/null 2>&1; then
+ echo "Error: git-lfs is required but not installed. Install it first: brew install git-lfs (macOS) or apt-get install git-lfs (Linux)" >&2
+ exit 1
+fi
+
+APPWORLD_ROOT="."
+export APPWORLD_ROOT
+
+TMPDIR="$(mktemp -d)"
+git lfs install >/dev/null 2>&1 || true
+git clone https://github.com/StonyBrookNLP/appworld.git "$TMPDIR/appworld"
+cd "$TMPDIR/appworld"
+git checkout edc960129fa6889c2b381715ecd108982029f6d1
+git lfs pull
+
+uv pip install "."
+
+python -m appworld.cli install
+
+cd - >/dev/null 2>&1 || true
+rm -rf "$TMPDIR"
+python -m appworld.cli download data --root "."
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/bfcl/__init__.py b/labs/AgentStream/exgentic/src/exgentic/benchmarks/bfcl/__init__.py
new file mode 100644
index 00000000..7f1e6dab
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/bfcl/__init__.py
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+"""BFCL benchmark adapter."""
+
+from .bfcl_benchmark import BFCLBenchmark
+
+__all__ = ["BFCLBenchmark"]
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/bfcl/bfcl_benchmark.py b/labs/AgentStream/exgentic/src/exgentic/benchmarks/bfcl/bfcl_benchmark.py
new file mode 100644
index 00000000..f804858d
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/bfcl/bfcl_benchmark.py
@@ -0,0 +1,93 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+"""BFCL benchmark adapter — light benchmark class only.
+
+Evaluator, session, and helper classes live in ``bfcl_eval.py``
+and are loaded inside the runner subprocess via ``_get_evaluator_class()``
+and ``_get_session_class()``. This file must remain importable without
+the ``bfcl_eval`` package installed.
+"""
+
+from __future__ import annotations
+
+from typing import Any, ClassVar, Literal
+
+from pydantic import BaseModel, ConfigDict
+
+from ...core import Benchmark
+from ...core.types import FinishAction
+
+BFCLSubset = Literal[
+ "simple_python",
+ "simple_java",
+ "simple_javascript",
+ "multiple",
+ "parallel",
+ "parallel_multiple",
+ "irrelevance",
+ "live_simple",
+ "live_multiple",
+ "live_parallel",
+ "live_parallel_multiple",
+ "live_irrelevance",
+ "live_relevance",
+ "multi_turn_base",
+ "multi_turn_long_context",
+ "multi_turn_miss_func",
+ "multi_turn_miss_param",
+]
+
+
+class BFCLFinishArgs(BaseModel):
+ content: str = ""
+
+
+class BFCLFinishAction(FinishAction):
+ name: Literal["finish"] = "finish"
+ arguments: BFCLFinishArgs
+
+
+class BFCLBenchmark(Benchmark, BaseModel):
+ """BFCL benchmark using Gorilla assets with an Exgentic-native runtime."""
+
+ display_name: ClassVar[str] = "BFCL"
+ slug_name: ClassVar[str] = "bfcl"
+ available_subsets: ClassVar[list[str]] = [
+ "simple_python",
+ "simple_java",
+ "simple_javascript",
+ "multiple",
+ "parallel",
+ "parallel_multiple",
+ "irrelevance",
+ "live_simple",
+ "live_multiple",
+ "live_parallel",
+ "live_parallel_multiple",
+ "live_irrelevance",
+ "live_relevance",
+ "multi_turn_base",
+ "multi_turn_long_context",
+ "multi_turn_miss_func",
+ "multi_turn_miss_param",
+ ]
+ model_config = ConfigDict(arbitrary_types_allowed=True, populate_by_name=True)
+
+ @classmethod
+ def _get_evaluator_class(cls):
+ return "exgentic.benchmarks.bfcl.bfcl_eval:BFCLEvaluator"
+
+ @classmethod
+ def _get_session_class(cls):
+ return "exgentic.benchmarks.bfcl.bfcl_eval:BFCLSession"
+
+ subset: BFCLSubset = "simple_python"
+
+ def list_subsets(self) -> list[str]: # type: ignore[override]
+ return list(self.available_subsets)
+
+ def _get_evaluator_kwargs(self) -> dict[str, Any]:
+ return {
+ "subset": self.subset,
+ }
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/bfcl/bfcl_eval.py b/labs/AgentStream/exgentic/src/exgentic/benchmarks/bfcl/bfcl_eval.py
new file mode 100644
index 00000000..46386610
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/bfcl/bfcl_eval.py
@@ -0,0 +1,576 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+"""BFCL evaluator and session classes.
+
+These classes import bfcl_eval (via bfcl_shim) at runtime. They are only
+ever instantiated inside the isolated runner subprocess, so the heavy
+``bfcl_eval`` dependency is never required in the host process.
+"""
+
+from __future__ import annotations
+
+import json
+from copy import deepcopy
+from pathlib import Path
+from typing import Any, Literal
+
+from pydantic import BaseModel
+
+from ...adapters.schemas.openai import openai_tools_to_action_types
+from ...core.evaluator import Evaluator
+from ...core.session import Session
+from ...core.types import (
+ Action,
+ ActionType,
+ BenchmarkResults,
+ EmptyObservation,
+ MultiObservation,
+ SessionIndex,
+ SessionScore,
+ SingleAction,
+ SingleObservation,
+)
+from .bfcl_benchmark import BFCLFinishAction
+from .bfcl_shim import load_bfcl_symbols
+
+BFCLSubset = Literal[
+ "simple_python",
+ "simple_java",
+ "simple_javascript",
+ "multiple",
+ "parallel",
+ "parallel_multiple",
+ "irrelevance",
+ "live_simple",
+ "live_multiple",
+ "live_parallel",
+ "live_parallel_multiple",
+ "live_irrelevance",
+ "live_relevance",
+ "multi_turn_base",
+ "multi_turn_long_context",
+ "multi_turn_miss_func",
+ "multi_turn_miss_param",
+]
+
+
+def _is_relevance_subset(subset: str) -> bool:
+ return subset in {"irrelevance", "live_irrelevance", "live_relevance"}
+
+
+def _is_multi_turn_subset(subset: str) -> bool:
+ return subset.startswith("multi_turn_")
+
+
+def _language_for_subset(subset: str, symbols: dict[str, Any]) -> Any:
+ language = symbols["Language"]
+ if subset == "simple_java":
+ return language.JAVA
+ if subset == "simple_javascript":
+ return language.JAVASCRIPT
+ return language.PYTHON
+
+
+def _merge_observations(
+ *items: SingleObservation | MultiObservation | None,
+) -> SingleObservation | MultiObservation | None:
+ observations: list[SingleObservation] = []
+ for item in items:
+ if item is None:
+ continue
+ if isinstance(item, MultiObservation):
+ observations.extend(item.observations)
+ continue
+ observations.append(item)
+
+ if not observations:
+ return None
+ if len(observations) == 1:
+ return observations[0]
+ return MultiObservation(observations=observations)
+
+
+def _action_arguments_dict(action: SingleAction) -> dict[str, Any]:
+ arguments = action.arguments
+ if isinstance(arguments, BaseModel):
+ return arguments.model_dump()
+ if isinstance(arguments, dict):
+ return dict(arguments)
+ return {"value": arguments}
+
+
+def _render_turn_text(turn_messages: list[dict[str, Any]]) -> str:
+ parts: list[str] = []
+ for message in turn_messages:
+ if not isinstance(message, dict):
+ continue
+ content = str(message.get("content", "")).strip()
+ if not content:
+ continue
+ role = str(message.get("role", ""))
+ if role == "system":
+ parts.append(f"System: {content}")
+ else:
+ parts.append(content)
+ return "\n\n".join(parts)
+
+
+def _write_json(path: Path, payload: dict[str, Any]) -> None:
+ path.parent.mkdir(parents=True, exist_ok=True)
+ with open(path, "w", encoding="utf-8") as handle:
+ json.dump(payload, handle, ensure_ascii=False, indent=2, default=str)
+
+
+class BFCLSession(Session):
+ """Exgentic-native BFCL session with Gorilla-backed scoring."""
+
+ def __init__(
+ self,
+ subset: BFCLSubset,
+ prompt_entry: dict[str, Any],
+ possible_answer_entry: dict[str, Any] | None,
+ session_id: str | None = None,
+ ) -> None:
+ if session_id is not None:
+ self._session_id = session_id
+
+ self.subset = subset
+ self.prompt_entry = prompt_entry
+ self.possible_answer_entry = possible_answer_entry
+ self._task_id = str(prompt_entry["id"])
+ self._conversation_turns = [list(turn) for turn in prompt_entry.get("question", [])]
+ self._turn_texts = [_render_turn_text(turn) for turn in self._conversation_turns]
+ self._task_text = self._turn_texts[0] if self._turn_texts else ""
+ self._action_types = self._build_action_types(prompt_entry)
+
+ self._current_turn_index = 0
+ self._completed = False
+ self._result_payload: dict[str, Any] | None = None
+
+ turn_count = max(1, len(self._conversation_turns))
+ self._turn_step_calls: list[list[list[str]]] = [[] for _ in range(turn_count)]
+ self._turn_step_actions: list[list[list[dict[str, Any]]]] = [[] for _ in range(turn_count)]
+
+ super().__init__()
+
+ @property
+ def task(self) -> str:
+ return self._task_text
+
+ @property
+ def context(self) -> dict[str, Any]:
+ return {"policy": self._build_context_text()}
+
+ @property
+ def actions(self) -> list[ActionType]:
+ return self._action_types
+
+ @property
+ def task_id(self) -> str:
+ return self._task_id
+
+ def start(self) -> EmptyObservation:
+ return EmptyObservation()
+
+ def step(self, action: Action) -> SingleObservation | MultiObservation | None:
+ if self._completed:
+ return None
+ if action is None:
+ raise ValueError("BFCL requires an action or the dedicated finish action.")
+
+ flat_actions = action.to_action_list()
+ finish_requested = False
+ finish_action: SingleAction | None = None
+ tool_actions: list[SingleAction] = []
+ for item in flat_actions:
+ if item.name == "finish":
+ finish_requested = True
+ finish_action = item
+ continue
+ tool_actions.append(item)
+
+ step_call_strings = [self._action_to_function_call_string(item) for item in tool_actions]
+ if step_call_strings:
+ self._turn_step_calls[self._current_turn_index].append(step_call_strings)
+ self._turn_step_actions[self._current_turn_index].append(
+ [self._serialize_action(item) for item in tool_actions]
+ )
+
+ outputs = self._build_step_output_observation(tool_actions, step_call_strings)
+ if finish_requested:
+ # Generate a tool result for the finish action so that every
+ # tool_call_id in the assistant message has a matching tool
+ # response. Without this, providers with strict message
+ # validation (e.g. Azure OpenAI) reject the next request.
+ finish_obs = SingleObservation(
+ result="Turn finished.",
+ invoking_actions=[finish_action],
+ )
+ outputs = _merge_observations(outputs, finish_obs) if outputs else finish_obs
+ return self._finish_turn(outputs)
+
+ return outputs or EmptyObservation()
+
+ def done(self) -> bool:
+ return self._completed
+
+ def score(self) -> SessionScore:
+ if self._result_payload is None:
+ self._result_payload = self._compute_result_payload()
+
+ return SessionScore(
+ score=float(self._result_payload["score"]),
+ success=bool(self._result_payload["success"]),
+ is_finished=self._result_payload.get("is_finished"),
+ session_metrics=self._result_payload.get("session_metrics", {}),
+ session_metadata=self._result_payload.get("session_metadata", {}),
+ )
+
+ def close(self) -> None:
+ if self._result_payload is None:
+ self._result_payload = self._compute_result_payload()
+ _write_json(self.paths.benchmark_results, self._result_payload)
+
+ def _build_action_types(self, prompt_entry: dict[str, Any]) -> list[ActionType]:
+ symbols = load_bfcl_symbols()
+ functions = self._collect_functions(prompt_entry)
+ openai_tools = symbols["convert_to_tool"](
+ functions,
+ symbols["GORILLA_TO_OPENAPI"],
+ symbols["ModelStyle"].OPENAI_COMPLETIONS,
+ )
+ action_types = openai_tools_to_action_types(openai_tools)
+ action_types.append(
+ ActionType(
+ name="finish",
+ description="End the current BFCL step.",
+ cls=BFCLFinishAction,
+ is_finish=True,
+ )
+ )
+ return action_types
+
+ def _collect_functions(self, prompt_entry: dict[str, Any]) -> list[dict[str, Any]]:
+ functions = [deepcopy(item) for item in prompt_entry.get("function", [])]
+ for items in prompt_entry.get("missed_function", {}).values():
+ for item in items:
+ functions.append(deepcopy(item))
+
+ deduped: list[dict[str, Any]] = []
+ seen: set[str] = set()
+ for item in functions:
+ name = str(item.get("name", ""))
+ if not name or name in seen:
+ continue
+ seen.add(name)
+ deduped.append(item)
+ return deduped
+
+ def _build_context_text(self) -> str:
+ if len(self._conversation_turns) <= 1:
+ return (
+ "Complete the task using one or more actions, then call the dedicated "
+ "finish action. Clarification questions or any type of interaction with "
+ "the user is not permitted. In this task, actions are recorded rather "
+ "than executed through a live environment. Calling finish ends the "
+ "execution."
+ )
+ if _is_multi_turn_subset(self.subset):
+ return (
+ "Complete the current step using one or more actions, then call the "
+ "dedicated finish action. The finish action ends the current turn only, "
+ "not the entire session — continue using tools in subsequent turns. "
+ "Clarification questions or any type of interaction with the user is "
+ "not permitted."
+ )
+ return (
+ "Complete the current step using one or more actions, then call the "
+ "dedicated finish action. Clarification questions or any type of "
+ "interaction with the user is not permitted. In this task, actions are "
+ "recorded rather than executed through a live environment."
+ )
+
+ def _build_turn_observation(self, turn_index: int) -> SingleObservation:
+ if turn_index >= len(self._turn_texts):
+ return EmptyObservation()
+ text = self._turn_texts[turn_index]
+ if not text:
+ return EmptyObservation()
+ return SingleObservation(
+ result=text,
+ invoking_actions=[],
+ )
+
+ def _finish_turn(
+ self, outputs: SingleObservation | MultiObservation | None
+ ) -> SingleObservation | MultiObservation | None:
+ if self._current_turn_index >= len(self._conversation_turns) - 1:
+ self._completed = True
+ return outputs
+
+ self._current_turn_index += 1
+ next_turn = self._build_turn_observation(self._current_turn_index)
+ return _merge_observations(outputs, next_turn) or EmptyObservation()
+
+ def _build_step_output_observation(
+ self,
+ tool_actions: list[SingleAction],
+ step_call_strings: list[str],
+ ) -> SingleObservation | MultiObservation | None:
+ if not tool_actions:
+ return None
+
+ if _is_multi_turn_subset(self.subset):
+ raw_results = self._execute_multi_turn_step(step_call_strings)
+ else:
+ raw_results = ["Action recorded." for _ in tool_actions]
+
+ observations = [
+ SingleObservation(result=result, invoking_actions=[action])
+ for action, result in zip(tool_actions, raw_results, strict=False)
+ ]
+ return _merge_observations(*observations)
+
+ def _execute_multi_turn_step(self, step_call_strings: list[str]) -> list[str]:
+ symbols = load_bfcl_symbols()
+ execution_results, _ = symbols["execute_multi_turn_func_call"](
+ func_call_list=step_call_strings,
+ initial_config=self.prompt_entry["initial_config"],
+ involved_classes=self.prompt_entry["involved_classes"],
+ model_name=f"{symbols['proxy_model_name']}_{self.session_id}_runtime",
+ test_entry_id=self._task_id,
+ long_context=("long_context" in self.subset),
+ is_evaL_run=False,
+ )
+ return execution_results
+
+ def _action_to_function_call_string(self, action: SingleAction) -> str:
+ arguments = _action_arguments_dict(action)
+ if not arguments:
+ return f"{action.name}()"
+ rendered = ", ".join(f"{key}={value!r}" for key, value in arguments.items())
+ return f"{action.name}({rendered})"
+
+ def _serialize_action(self, action: SingleAction) -> dict[str, Any]:
+ return {
+ "id": action.id,
+ "name": action.name,
+ "arguments": _action_arguments_dict(action),
+ }
+
+ def _flatten_semantic_actions(self) -> list[dict[str, Any]]:
+ flattened: list[dict[str, Any]] = []
+ for turn in self._turn_step_actions:
+ for step in turn:
+ for action in step:
+ flattened.append({action["name"]: action["arguments"]})
+ return flattened
+
+ def _compute_result_payload(self) -> dict[str, Any]:
+ trace_payload = {
+ "task_id": self._task_id,
+ "subset": self.subset,
+ "turn_step_calls": self._turn_step_calls,
+ "turn_step_actions": self._turn_step_actions,
+ }
+ trace_path = self.paths.benchmark_dir / "bfcl_trace.json"
+ _write_json(trace_path, trace_payload)
+
+ if not self._completed:
+ payload = {
+ "score": 0.0,
+ "success": False,
+ "is_finished": False,
+ "session_metrics": {
+ "completed_turns": self._current_turn_index,
+ },
+ "session_metadata": {
+ "bfcl_task_id": self._task_id,
+ "trace_file": str(trace_path),
+ },
+ }
+ _write_json(self.paths.benchmark_dir / "bfcl_score.json", payload)
+ return payload
+
+ try:
+ if _is_multi_turn_subset(self.subset):
+ payload = self._score_multi_turn(trace_path)
+ elif _is_relevance_subset(self.subset):
+ payload = self._score_relevance(trace_path)
+ else:
+ payload = self._score_ast(trace_path)
+ except Exception as exc:
+ payload = {
+ "score": 0.0,
+ "success": False,
+ "is_finished": False,
+ "session_metadata": {
+ "bfcl_task_id": self._task_id,
+ "trace_file": str(trace_path),
+ "error": str(exc),
+ "error_source": "benchmark",
+ },
+ }
+
+ _write_json(self.paths.benchmark_dir / "bfcl_score.json", payload)
+ return payload
+
+ def _score_relevance(self, trace_path: Path) -> dict[str, Any]:
+ tool_call_count = len(self._flatten_semantic_actions())
+ success = tool_call_count == 0 if "irrelevance" in self.subset else tool_call_count > 0
+ return {
+ "score": 1.0 if success else 0.0,
+ "success": success,
+ "is_finished": True,
+ "session_metrics": {
+ "tool_call_count": tool_call_count,
+ "accuracy": 1.0 if success else 0.0,
+ },
+ "session_metadata": {
+ "bfcl_task_id": self._task_id,
+ "trace_file": str(trace_path),
+ },
+ }
+
+ def _score_ast(self, trace_path: Path) -> dict[str, Any]:
+ if self.possible_answer_entry is None:
+ raise ValueError(f"Missing ground truth for subset '{self.subset}'.")
+
+ symbols = load_bfcl_symbols()
+ checker_result = symbols["ast_checker"](
+ self.prompt_entry["function"],
+ self._flatten_semantic_actions(),
+ self.possible_answer_entry["ground_truth"],
+ _language_for_subset(self.subset, symbols),
+ self.subset,
+ symbols["proxy_model_name"],
+ )
+ score = 1.0 if checker_result["valid"] else 0.0
+ return {
+ "score": score,
+ "success": bool(checker_result["valid"]),
+ "is_finished": True,
+ "session_metrics": {
+ "accuracy": score,
+ "action_count": len(self._flatten_semantic_actions()),
+ },
+ "session_metadata": {
+ "bfcl_task_id": self._task_id,
+ "trace_file": str(trace_path),
+ "checker_result": checker_result,
+ },
+ }
+
+ def _score_multi_turn(self, trace_path: Path) -> dict[str, Any]:
+ if self.possible_answer_entry is None:
+ raise ValueError(f"Missing ground truth for subset '{self.subset}'.")
+
+ model_turns = self._turn_step_calls
+ ground_truth_turns = self.possible_answer_entry["ground_truth"]
+ if len(model_turns) != len(ground_truth_turns):
+ checker_result = {
+ "valid": False,
+ "error_message": (
+ "Model was force-terminated before completing all turns. "
+ f"Observed {len(model_turns)} turns for {len(ground_truth_turns)} ground-truth turns."
+ ),
+ "error_type": "multi_turn:force_terminated",
+ }
+ else:
+ symbols = load_bfcl_symbols()
+ checker_result = symbols["multi_turn_checker"](
+ model_turns,
+ ground_truth_turns,
+ self.prompt_entry,
+ self.subset,
+ f"{symbols['proxy_model_name']}_{self.session_id}_score",
+ )
+
+ score = 1.0 if checker_result["valid"] else 0.0
+ return {
+ "score": score,
+ "success": bool(checker_result["valid"]),
+ "is_finished": True,
+ "session_metrics": {
+ "accuracy": score,
+ "turn_count": len(model_turns),
+ },
+ "session_metadata": {
+ "bfcl_task_id": self._task_id,
+ "trace_file": str(trace_path),
+ "checker_result": checker_result,
+ },
+ }
+
+
+# ── Evaluator ────────────────────────────────────────────────────────
+
+
+class BFCLEvaluator(Evaluator):
+ """Evaluator for BFCL — task discovery, session kwargs, aggregation."""
+
+ def __init__(self, subset: str = "simple_python") -> None:
+ self._subset: BFCLSubset = subset # type: ignore[assignment]
+ self._entries: list[dict[str, Any]] | None = None
+ self._answers_by_id: dict[str, dict[str, Any]] | None = None
+
+ def _ensure_loaded(self) -> None:
+ if self._entries is not None and self._answers_by_id is not None:
+ return
+ symbols = load_bfcl_symbols()
+ entries = symbols["load_dataset_entry"](self._subset)
+ answers = (
+ {}
+ if _is_relevance_subset(self._subset)
+ else {str(entry["id"]): entry for entry in symbols["load_ground_truth_entry"](self._subset)}
+ )
+ self._entries = entries
+ self._answers_by_id = answers
+
+ def list_tasks(self) -> list[str]:
+ self._ensure_loaded()
+ assert self._entries is not None
+ return [str(entry["id"]) for entry in self._entries]
+
+ def get_session_kwargs(self, index: SessionIndex) -> dict[str, Any]:
+ self._ensure_loaded()
+ assert self._entries is not None
+ prompt_entry = next(
+ (entry for entry in self._entries if str(entry["id"]) == str(index.task_id)),
+ None,
+ )
+ if prompt_entry is None:
+ raise KeyError(f"Unknown BFCL task id '{index.task_id}' for subset '{self._subset}'.")
+ answer_entry = None if _is_relevance_subset(self._subset) else self._answers_by_id.get(str(index.task_id))
+ return {
+ "subset": self._subset,
+ "prompt_entry": prompt_entry,
+ "possible_answer_entry": answer_entry,
+ "session_id": index.session_id,
+ }
+
+ def aggregate_sessions(self, sessions: list[SessionIndex]) -> BenchmarkResults:
+ payloads: list[dict[str, Any]] = []
+ for paths in self.get_sessions_paths(sessions):
+ result_path = paths.benchmark_results
+ if not result_path.exists():
+ raise FileNotFoundError(
+ f"Missing BFCL results for planned session '{paths.session_id}' at {result_path}"
+ )
+ with open(result_path, encoding="utf-8") as handle:
+ payloads.append(json.load(handle))
+
+ total_tasks = len(payloads)
+ total_score = sum(float(payload.get("score", 0.0)) for payload in payloads)
+ successes = sum(1 for payload in payloads if payload.get("success"))
+ return BenchmarkResults(
+ benchmark_name=f"bfcl-{self._subset}",
+ total_tasks=total_tasks,
+ score=(total_score / total_tasks) if total_tasks else 0.0,
+ metrics={
+ "subset": self._subset,
+ "success_count": successes,
+ "accuracy": (total_score / total_tasks) if total_tasks else 0.0,
+ },
+ )
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/bfcl/bfcl_shim.py b/labs/AgentStream/exgentic/src/exgentic/benchmarks/bfcl/bfcl_shim.py
new file mode 100644
index 00000000..88fcfafc
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/bfcl/bfcl_shim.py
@@ -0,0 +1,68 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+"""Thin import shim around Gorilla's BFCL package."""
+
+from __future__ import annotations
+
+from functools import lru_cache
+from typing import Any
+
+EXGENTIC_BFCL_MODEL = "exgentic-proxy-fc"
+
+
+@lru_cache(maxsize=1)
+def load_bfcl_symbols() -> dict[str, Any]:
+ from bfcl_eval.constants.enums import Language, ModelStyle
+ from bfcl_eval.constants.model_config import MODEL_CONFIG_MAPPING, ModelConfig
+ from bfcl_eval.constants.type_mappings import GORILLA_TO_OPENAPI
+ from bfcl_eval.eval_checker.ast_eval.ast_checker import ast_checker
+ from bfcl_eval.eval_checker.eval_runner import (
+ ast_file_runner,
+ multi_turn_runner,
+ relevance_file_runner,
+ )
+ from bfcl_eval.eval_checker.multi_turn_eval.multi_turn_checker import (
+ multi_turn_checker,
+ multi_turn_irrelevance_checker,
+ )
+ from bfcl_eval.eval_checker.multi_turn_eval.multi_turn_utils import (
+ execute_multi_turn_func_call,
+ )
+ from bfcl_eval.model_handler.api_inference.openai_completion import OpenAICompletionsHandler
+ from bfcl_eval.model_handler.base_handler import BaseHandler
+ from bfcl_eval.model_handler.utils import convert_to_tool
+ from bfcl_eval.utils import load_dataset_entry, load_ground_truth_entry
+
+ if EXGENTIC_BFCL_MODEL not in MODEL_CONFIG_MAPPING:
+ MODEL_CONFIG_MAPPING[EXGENTIC_BFCL_MODEL] = ModelConfig(
+ model_name=EXGENTIC_BFCL_MODEL,
+ display_name="Exgentic Proxy (FC)",
+ url="https://github.com/Exgentic/exgentic",
+ org="Exgentic",
+ license="Apache 2.0",
+ model_handler=OpenAICompletionsHandler,
+ input_price=None,
+ output_price=None,
+ is_fc_model=True,
+ underscore_to_dot=True,
+ )
+
+ return {
+ "BaseHandler": BaseHandler,
+ "Language": Language,
+ "ModelStyle": ModelStyle,
+ "OpenAICompletionsHandler": OpenAICompletionsHandler,
+ "GORILLA_TO_OPENAPI": GORILLA_TO_OPENAPI,
+ "convert_to_tool": convert_to_tool,
+ "load_dataset_entry": load_dataset_entry,
+ "load_ground_truth_entry": load_ground_truth_entry,
+ "ast_checker": ast_checker,
+ "multi_turn_checker": multi_turn_checker,
+ "multi_turn_irrelevance_checker": multi_turn_irrelevance_checker,
+ "execute_multi_turn_func_call": execute_multi_turn_func_call,
+ "ast_file_runner": ast_file_runner,
+ "multi_turn_runner": multi_turn_runner,
+ "relevance_file_runner": relevance_file_runner,
+ "proxy_model_name": EXGENTIC_BFCL_MODEL,
+ }
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/bfcl/setup.sh b/labs/AgentStream/exgentic/src/exgentic/benchmarks/bfcl/setup.sh
new file mode 100644
index 00000000..3af65e29
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/bfcl/setup.sh
@@ -0,0 +1,30 @@
+#!/bin/bash
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+INSTALL_ROOT="${SCRIPT_DIR}/installation"
+REPO_DIR="${INSTALL_ROOT}/gorilla"
+BFCL_DIR="${REPO_DIR}/berkeley-function-call-leaderboard"
+GORILLA_URL="https://github.com/ShishirPatil/gorilla.git"
+GORILLA_REF="7ad0134c665944819f88bc50862108d94015968b" # pragma: allowlist secret
+
+pip_install() {
+ if command -v uv >/dev/null 2>&1; then
+ uv pip install "$@"
+ else
+ python -m pip install "$@"
+ fi
+}
+
+mkdir -p "${INSTALL_ROOT}"
+
+if [ ! -d "${REPO_DIR}/.git" ]; then
+ rm -rf "${REPO_DIR}"
+ git clone "${GORILLA_URL}" "${REPO_DIR}"
+fi
+
+git -C "${REPO_DIR}" fetch --depth 1 origin "${GORILLA_REF}"
+git -C "${REPO_DIR}" checkout --force "${GORILLA_REF}"
+
+pip_install -e "${BFCL_DIR}"
+pip_install soundfile
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/__init__.py b/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/__init__.py
new file mode 100644
index 00000000..367fec41
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/__init__.py
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/browsecomp_benchmark.py b/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/browsecomp_benchmark.py
new file mode 100644
index 00000000..b8c4eaaf
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/browsecomp_benchmark.py
@@ -0,0 +1,697 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from __future__ import annotations
+
+import datetime
+import json
+import os
+from collections import defaultdict
+from collections.abc import Mapping
+from pathlib import Path
+from typing import TYPE_CHECKING, Any, ClassVar, Literal, Optional
+
+from pydantic import BaseModel, ConfigDict, Field
+
+from ...core import Benchmark, Session
+from ...core.actions import ActionsHandler
+from ...core.evaluator import Evaluator
+from ...core.types import (
+ Action,
+ ActionType,
+ BenchmarkResults,
+ EmptyObservation,
+ FinishAction,
+ MessageAction,
+ Observation,
+ SessionIndex,
+ SessionScore,
+ SingleAction,
+ SingleObservation,
+)
+from ...environment.instance import get_manager
+from ...utils.cost import CostReport, LiteLLMCostReport
+from ...utils.settings import RunnerName
+from .retriever import RetrieverClient, get_retriever_url, get_shared_retriever
+
+if TYPE_CHECKING:
+ from .browsecomp_eval import BrowseCompEvaluator
+
+# Paper-reported total for the full BrowseCompPlus dataset.
+DEFAULT_TOTAL_TASKS = 830
+
+
+class BrowseCompPlusSearchArgs(BaseModel):
+ query: str
+
+
+class BrowseCompPlusSearchAction(SingleAction):
+ name: Literal["search"] = "search"
+ arguments: BrowseCompPlusSearchArgs
+
+
+class BrowseCompPlusGetDocumentsArgs(BaseModel):
+ docid: str
+
+
+class BrowseCompPlusGetDocAction(SingleAction):
+ name: Literal["get_document"] = "get_document"
+ arguments: BrowseCompPlusGetDocumentsArgs
+
+
+class BrowseCompPlusFinishArgs(BaseModel):
+ exact_answer: str = Field(description="Your succinct, final answer")
+ explanation: str = Field(
+ description=(
+ "Your explanation for your final answer. For this"
+ " explanation section only, you should cite your"
+ " evidence documents inline by enclosing their docids"
+ " in square brackets [] at the end of sentences."
+ " For example, [20]."
+ )
+ )
+ confidence: float = Field(description="Your confidence score between 0% and 100% for your answer")
+
+
+class BrowseCompPlusFinishAction(FinishAction):
+ name: Literal["submit"] = "submit"
+ arguments: BrowseCompPlusFinishArgs
+
+
+class BrowseCompPlusSession(Session):
+ _done: bool
+ evaluator: BrowseCompEvaluator
+
+ def __init__(
+ self,
+ instance: dict[str, Any],
+ searcher_params: dict[str, Any],
+ eval_model_id: str = "openai/Azure/gpt-4.1",
+ max_interactions: int | None = 100,
+ session_id: str | None = None,
+ retriever_url: str | None = None,
+ **_kwargs: Any,
+ ) -> None:
+ if session_id is not None:
+ self._session_id = session_id
+ self._instance = instance.copy()
+ self._task_id = instance["task_id"]
+ self._done = False
+ self._searcher_params = searcher_params
+
+ # Initialize search: use shared retriever service if URL provided,
+ # otherwise load the index locally via SearchService singleton.
+ if retriever_url:
+ self._init_retriever_client(retriever_url, searcher_params)
+ else:
+ self._init_search_tool_handler(searcher_params)
+
+ self._registry = ActionsHandler(logger=self.logger)
+ self.set_action_types()
+ self._response = None
+ self.evaluator = self.get_evaluator(eval_model_id)
+ self.retrieved_docids = set()
+ self.tool_call_count = defaultdict(lambda: 0)
+ self.model_usage = None
+ self.total_actions_executed = 0
+ self.max_interactions = max_interactions
+ self.logger.info(f"Running for query id {self.task_id}")
+ super().__init__()
+
+ def _init_retriever_client(self, url: str, searcher_params: dict[str, Any]) -> None:
+ """Connect to a shared Retriever service by URL."""
+ from .search_tool_handler import BCPSearchToolHandler
+
+ client = RetrieverClient(url)
+ self.search_tool_handler = BCPSearchToolHandler(
+ searcher=client,
+ snippet_max_tokens=searcher_params["max_snippet_length"],
+ k=searcher_params["top_k_docs"],
+ include_get_document=searcher_params["include_get_document"],
+ full_doc_max_tokens=searcher_params["full_doc_max_tokens"],
+ )
+
+ def _init_search_tool_handler(self, searcher_params: dict[str, Any]) -> None:
+ """Initialize search tool handler using local singleton service."""
+ from searcher.searchers import SearcherType
+
+ from .search_service import get_search_service
+ from .search_tool_handler import BCPSearchToolHandler
+
+ # Get searcher from singleton service
+ search_service = get_search_service()
+ searcher_class = SearcherType.get_searcher_class(searcher_params["searcher_type"])
+
+ # Extract searcher-specific args
+ searcher_args = {
+ k: v
+ for k, v in searcher_params.items()
+ if k
+ not in [
+ "searcher_type",
+ "max_snippet_length",
+ "top_k_docs",
+ "include_get_document",
+ "full_doc_max_tokens",
+ ]
+ }
+
+ # Pass session's logger to the service
+ searcher_obj = search_service.get_or_create_searcher(
+ searcher_type=searcher_params["searcher_type"],
+ searcher_class=searcher_class,
+ logger=self.logger,
+ **searcher_args,
+ )
+
+ # Create search tool handler
+ self.search_tool_handler = BCPSearchToolHandler(
+ searcher=searcher_obj,
+ snippet_max_tokens=searcher_params["max_snippet_length"],
+ k=searcher_params["top_k_docs"],
+ include_get_document=searcher_params["include_get_document"],
+ full_doc_max_tokens=searcher_params["full_doc_max_tokens"],
+ )
+
+ @property
+ def task(self) -> str:
+ get_doc_str = "and document expansion " if self.search_tool_handler.include_get_document else ""
+ query = self._instance["query"]
+ return (
+ "Answer the provided question by performing search "
+ f"{get_doc_str}as needed, and submit your final"
+ " answer.\n"
+ f"Question: {query}\n"
+ "Note:\n"
+ "- The question has an answer discoverable through"
+ " proper search.\n"
+ "- The question requires putting together information"
+ " from different sources.\n"
+ "\n"
+ "Your performance is scored based on:\n"
+ " 1. Most importantly, the correctness of the"
+ " answer you assembled from different searches.\n"
+ " 2. Your effective use of search and your ability"
+ " to retrieve all relevant information for the"
+ " question.\n"
+ " 3. How efficiently you find all the relevant"
+ " information, using as few searches as possible.\n"
+ "\n"
+ "Important: During your work, Do NOT interact with"
+ " the user or send any messages at any point"
+ " -- messages will be ignored and are NOT considered"
+ " a valid final answer. The ONLY acceptable way to"
+ " finish is by calling 'submit' with the required"
+ " structured fields.\n"
+ "\n"
+ "Finish the session always by calling `submit`."
+ " If you fail to find the answer, submit with"
+ ' exact_answer: "Can\'t find the answer.".'
+ )
+
+ @property
+ def context(self) -> dict[str, Any]:
+ return {}
+
+ @property
+ def actions(self) -> list[ActionType]:
+ return self._registry.actions
+
+ @property
+ def task_id(self) -> str:
+ """Task identifier."""
+ return str(self._instance["query_id"])
+
+ def _to_observation(self, raw: Any, invoking: list[SingleAction] | None = None) -> Observation:
+ return SingleObservation(invoking_actions=invoking or [], result=raw)
+
+ def start(self):
+ return EmptyObservation()
+
+ def record_single_action(self, action: SingleAction) -> None:
+ self.logger.info(f"Received *{action.name}* action with arguments: {action.arguments}")
+ self.tool_call_count[action.name] += 1
+ self.total_actions_executed += 1
+
+ def step(self, action: Action) -> Optional[Observation]:
+ if action is None:
+ self._done = True
+
+ if self.total_actions_executed >= self.max_interactions:
+ self.logger.info(f"Reached maximal limit of {self.total_actions_executed} allowed actions")
+ self._done = True
+
+ if self._done:
+ return None
+
+ observation = self._registry.execute(action)
+ return observation
+
+ # Action handlers ------------------------------------------------------------
+ def _handle_search(self, action: SingleAction) -> Any:
+ self.record_single_action(action)
+ result = self.get_search_result(action)
+ # keep retrieved docs
+ self.record_retrieved_docids(result)
+ return result
+
+ def _handle_get_document(self, action: SingleAction) -> SingleAction | None:
+ self.record_single_action(action)
+ args_dict = self.get_arguments_dict(action.arguments)
+ return self.search_tool_handler.execute_tool("get_document", args_dict)
+
+ def _handle_finish(self, action: SingleAction) -> SingleAction | None:
+ self.record_single_action(action)
+ final_response = self.get_arguments_dict(action.arguments)
+ self._response = json.dumps(final_response)
+ self._done = True
+ return None
+
+ def done(self) -> bool:
+ return self._done
+
+ def score(self) -> SessionScore:
+ results, self.model_usage = self.evaluator.evaluate_response(
+ agent_response=self._response,
+ instance=self._instance,
+ retrieved_docids_set=self.retrieved_docids,
+ tool_call_counts=self.tool_call_count,
+ )
+ return results
+
+ def get_cost(self) -> CostReport:
+ if not self.model_usage:
+ return LiteLLMCostReport.initialize_empty(model_name=self.evaluator.eval_model_id)
+
+ return LiteLLMCostReport.from_token_counts(
+ model_name=self.evaluator.eval_model_id,
+ input_tokens=self.model_usage["prompt_tokens"],
+ output_tokens=self.model_usage["completion_tokens"],
+ )
+
+ def close(self) -> None:
+ super().close()
+ # Persist minimal results for aggregation
+ sc = self.score()
+ self.save_results(sc.model_dump())
+
+ def set_action_types(self):
+ k = self.search_tool_handler.k
+ n_tokens = self.search_tool_handler.snippet_max_tokens
+ self._registry.add_action(
+ name="search",
+ description=(
+ "Perform a search on a knowledge source: supply"
+ " a single 'query' string; the action retrieves"
+ f" the {k} top most relevant results, each"
+ f" trimmed to {n_tokens} tokens."
+ ),
+ action_cls=BrowseCompPlusSearchAction,
+ handler=self._handle_search,
+ )
+ self._registry.add_action(
+ name="submit",
+ description="Submit final answer and complete",
+ action_cls=BrowseCompPlusFinishAction,
+ handler=self._handle_finish,
+ is_finish=True,
+ )
+
+ if self.search_tool_handler.include_get_document:
+ self._registry.add_action(
+ name="get_document",
+ description="Retrieve the full document using its document id",
+ action_cls=BrowseCompPlusGetDocAction,
+ handler=self._handle_get_document,
+ )
+ # todo: agents fail in practice without this option, even with explicit instructions. should we keep it?
+ self._registry.add_action(
+ name="message",
+ description="Send the final answer as a message to the user",
+ action_cls=MessageAction,
+ handler=self._handle_finish,
+ is_hidden=True,
+ is_message=True,
+ )
+
+ def get_evaluator(self, eval_model_id):
+ from .browsecomp_eval import BrowseCompEvaluatorOpenai, BrowsecompEvaluatorQwen
+
+ if "gpt" in eval_model_id:
+ return BrowseCompEvaluatorOpenai(eval_model_id=eval_model_id)
+ if eval_model_id == "Qwen/Qwen3-32B":
+ return BrowsecompEvaluatorQwen() # Currently not supported
+ raise ValueError(f"Invalid eval_model_id: {eval_model_id}")
+
+ # Robustly extract the answer from pydantic model or dict
+ def get_arguments_dict(self, args):
+ if isinstance(args, BaseModel):
+ return args.model_dump() # Pydantic v2
+ if isinstance(args, Mapping):
+ return dict(args)
+ return str(getattr(args, "value", {}))
+
+ def record_retrieved_docids(self, result: str):
+ try:
+ result = json.loads(result)
+ retrieved_docids = {result.get("docid") for result in result}
+ self.retrieved_docids = self.retrieved_docids | retrieved_docids
+ except json.decoder.JSONDecodeError:
+ self.logger.error(f"Failed to retrieve docids: {result}")
+
+ def get_search_result(self, act):
+ from .searcher_cache import SearchDiskCacheSession
+
+ args_dict = self.get_arguments_dict(act.arguments)
+ searcher_params = self.get_searcher_params()
+ search_cache = SearchDiskCacheSession(args_dict["query"], **searcher_params)
+ search_cache.logger = self.logger
+ result = search_cache.handle_start_fetch_results()
+ if result:
+ self.logger.info(f"Retrieved docs from cache: {result}")
+ else:
+ result = self.search_tool_handler.execute_tool(act.name, args_dict)
+ search_cache.cache_results(result)
+ self.logger.info(f"Retrieved docs from searcher: {result}")
+ return result
+
+ def get_searcher_params(self):
+ return {
+ "n": self.search_tool_handler.snippet_max_tokens,
+ "k": self.search_tool_handler.k,
+ "search_type": self._searcher_params.get("searcher_type", "unknown"),
+ "search_model": self._searcher_params.get("model_name"),
+ "normalize": self._searcher_params.get("normalize"),
+ }
+
+
+# ── Evaluator ────────────────────────────────────────────────────────
+
+
+class BrowseCompPlusEvaluator(Evaluator):
+ """Evaluator for BrowseCompPlus — task discovery, session kwargs, aggregation."""
+
+ def __init__(
+ self,
+ subset: str = "main",
+ searcher_type: str = "faiss",
+ searcher_model_name: str = "Qwen/Qwen3-Embedding-8B",
+ max_snippet_length: int = 512,
+ top_k_docs: int = 5,
+ include_get_document: bool = True,
+ normalize_search: bool = True,
+ full_doc_max_tokens: int = 2048,
+ max_interactions: int | None = 100,
+ inference_model: str = "N/A",
+ retriever_url: str | None = None,
+ eval_model_id: str = "openai/Azure/gpt-4.1",
+ ) -> None:
+ self._subset = subset
+ self._searcher_type = searcher_type
+ self._searcher_model_name = searcher_model_name
+ self._max_snippet_length = max_snippet_length
+ self._top_k_docs = top_k_docs
+ self._include_get_document = include_get_document
+ self._normalize_search = normalize_search
+ self._full_doc_max_tokens = full_doc_max_tokens
+ self._max_interactions = max_interactions
+ self._inference_model = inference_model
+ self._retriever_url = retriever_url
+ self._eval_model_id = eval_model_id
+ self._dataset: list[dict[str, Any]] | None = None
+ self._task_lookup: dict[str, dict[str, Any]] | None = None
+
+ @property
+ def assets_dir(self):
+ return get_manager().env_path("benchmarks/browsecompplus")
+
+ def extract_dataset(self):
+ import pandas as pd
+
+ data_path = self.assets_dir / "data" / "browsecomp_plus_decrypted_docids.jsonl"
+ if not data_path.exists():
+ raise Exception(f"{data_path} does not exist. Run 'exgentic install --benchmark browsecompplus' first.")
+ instances = pd.read_json(path_or_buf=data_path, lines=True).to_dict(orient="records")
+
+ def proces_instance(instance):
+ processed_instance = {
+ "query_id": instance["query_id"],
+ "query": instance["query"],
+ "gold_answer": instance["answer"],
+ }
+ for k in ["gold_docs", "evidence_docs", "negative_docs"]:
+ processed_instance[k] = instance[k]
+ return processed_instance
+
+ instances = [proces_instance(instance) for instance in instances]
+ return instances
+
+ def _ensure_dataset(self) -> None:
+ if self._dataset is None:
+ self._dataset = self.extract_dataset()
+ self._task_lookup = {str(item["query_id"]): item for item in self._dataset}
+
+ def list_tasks(self) -> list[str]:
+ self._ensure_dataset()
+ return [str(item["query_id"]) for item in self._dataset]
+
+ def _get_searcher_params(self) -> dict[str, Any]:
+ """Get searcher parameters to pass to session."""
+ index_dir = self.assets_dir / "indexes"
+ if self._searcher_type == "bm25":
+ index_dir = index_dir / "bm25"
+ searcher_args = {"index_path": str(index_dir)}
+ else:
+ model_dir_name = self._searcher_model_name.lower().split("/")[-1]
+ index_dir = index_dir / model_dir_name
+ if not os.path.exists(index_dir):
+ available_models = os.listdir(self.assets_dir / "indexes")
+ raise FileNotFoundError(
+ f"Index dir for BrowseCompPlus benchmark {index_dir} does not exist. "
+ f"Please select an available embedding model out of {available_models}"
+ )
+ index_path = index_dir / "corpus.shard*_of_4.pkl"
+ searcher_args = {
+ "index_path": str(index_path),
+ "model_name": self._searcher_model_name,
+ "normalize": self._normalize_search,
+ }
+
+ return {
+ "searcher_type": self._searcher_type,
+ "max_snippet_length": self._max_snippet_length,
+ "top_k_docs": self._top_k_docs,
+ "include_get_document": self._include_get_document,
+ "full_doc_max_tokens": self._full_doc_max_tokens,
+ **searcher_args,
+ }
+
+ def get_session_kwargs(self, index: SessionIndex) -> dict[str, Any]:
+ self._ensure_dataset()
+ task_id = index.task_id
+ if self._task_lookup is None or task_id not in self._task_lookup:
+ raise KeyError(f"Unknown BrowseCompPlus task id '{task_id}'.")
+ instance = {"task_id": task_id, **self._task_lookup[task_id]}
+ kwargs: dict[str, Any] = {
+ "instance": instance,
+ "searcher_params": self._get_searcher_params(),
+ "max_interactions": self._max_interactions,
+ "session_id": index.session_id,
+ "eval_model_id": self._eval_model_id,
+ }
+ if self._retriever_url:
+ kwargs["retriever_url"] = self._retriever_url
+ return kwargs
+
+ def aggregate_sessions(self, sessions: list[SessionIndex]) -> BenchmarkResults:
+ # Aggregate per-session scores written by sessions
+ scores: list[float] = []
+ retrieval_recalls: list[float] = []
+ confidence_list: list[float] = []
+ tool_call_counts_list: list[dict[str, float]] = []
+ correctness: list[float] = []
+ for paths in self.get_sessions_paths(sessions):
+ fp = paths.benchmark_results
+ if not fp.exists():
+ raise FileNotFoundError(f"Missing results for planned session '{paths.session_id}' at {fp}")
+
+ with open(fp, encoding="utf-8-sig") as f:
+ payload = json.load(f)
+ if not payload:
+ raise ValueError(f"Empty benchmark results for session '{paths.session_id}' at {fp}")
+
+ s = float(payload["score"]) # minimal: assume exists
+ scores.append(s)
+ metrics = payload.get("session_metrics", {})
+ if not metrics:
+ print(f"No metrics for session '{paths.session_id}' at {fp}")
+
+ retrieval_recall = metrics.get("Retrieval_recall", 0)
+ retrieval_recalls.append(float(retrieval_recall) if retrieval_recall is not None else 0)
+ correctness.append(payload.get("success", 0))
+
+ confidence = metrics.get("Confidence")
+ try:
+ confidence = float(confidence)
+ except Exception:
+ confidence = 0
+ confidence_list.append(confidence)
+
+ metadata = payload.get("session_metadata", {})
+ if not metadata:
+ print(f"No metadata for session '{paths.session_id}' at {fp}")
+ tool_call_counts_list.append(metadata.get("tool_call_counts", {}))
+
+ avg = sum(scores) / len(scores) if scores else 0.0
+ avg_retrieval_recalls = sum(retrieval_recalls) / len(retrieval_recalls) if retrieval_recalls else 0.0
+ tools_keys = set().union(*tool_call_counts_list)
+ avg_tool_use_counts = {
+ k: sum(d.get(k, 0) for d in tool_call_counts_list) / len(tool_call_counts_list) for k in tools_keys
+ }
+
+ calibration_error = None
+ # calibration error only comupted for a large number of examples
+ if len(correctness) >= 100:
+ try:
+ from scripts_evaluation.evaluate_with_openai import calculate_calibration_error
+
+ calibration_error = calculate_calibration_error(confidences=confidence_list, correctness=correctness)
+ except Exception:
+ print(f"Failed to calculate calibration error for session '{paths.session_id}' at {fp}")
+ calibration_error = 0
+ metrics = {
+ "LLM": self._inference_model,
+ "Accuracy (%)": avg,
+ "Recall (%)": avg_retrieval_recalls,
+ "avg_tool_stats": avg_tool_use_counts,
+ "Calibration Error (%)": calibration_error,
+ "Retriever": self._searcher_model_name,
+ "Link": "change me when submitting",
+ "Evaluation Date": datetime.datetime.now().date().isoformat(),
+ }
+ return BenchmarkResults(
+ benchmark_name="BrowseCompPlus",
+ total_tasks=len(sessions),
+ score=avg,
+ metrics=metrics,
+ )
+
+
+# ── Benchmark config ─────────────────────────────────────────────────
+
+
+class BrowseCompPlusBenchmark(Benchmark, BaseModel):
+ display_name: ClassVar[str] = "BrowseCompPlus"
+ slug_name: ClassVar[str] = "browsecompplus"
+ model_config = ConfigDict(arbitrary_types_allowed=True)
+
+ @classmethod
+ def _get_evaluator_class(cls):
+ return BrowseCompPlusEvaluator
+
+ @classmethod
+ def _get_session_class(cls):
+ return BrowseCompPlusSession
+
+ subset: Literal["main"] = "main"
+
+ runner: RunnerName | None = None # Threadsafe; uses global default runner (venv)
+
+ # Retriever runner — when set, the search index runs as a shared service
+ # instead of being loaded in each session process. Useful when sessions
+ # run in Docker to avoid duplicating the heavy index in RAM.
+ retriever_runner: RunnerName | None = None
+
+ # Agent inference params (for logging)
+ inference_model: str = "N/A"
+
+ # searcher params
+ searcher_type: str = "faiss" # "bm25" or "faiss"
+ searcher_model_name: str = "Qwen/Qwen3-Embedding-8B" # Used for faiss only
+ max_snippet_length: int = 512
+ top_k_docs: int = 5
+ include_get_document: bool = True
+ normalize_search: bool = True
+ full_doc_max_tokens: int = 2048
+ max_interactions: int | None = 100
+ eval_model_id: str = "openai/Azure/gpt-4.1"
+
+ @property
+ def _assets_dir(self) -> str:
+ return str(get_manager().env_path("benchmarks/browsecompplus"))
+
+ def _retriever_runner_kwargs(self) -> dict[str, Any]:
+ """Runner kwargs for the retriever container (volumes).
+
+ Only returns Docker-specific kwargs when the retriever actually runs
+ in Docker; for 'service' or 'direct' these would leak into the
+ target class constructor and cause errors.
+ """
+ if self.retriever_runner != "docker":
+ return {}
+ kw: dict[str, Any] = {
+ "env_name": f"benchmarks/{self.slug_name}",
+ "module_path": type(self).__module__,
+ }
+ kw["volumes"] = {self._assets_dir: self._assets_dir}
+ return kw
+
+ def _get_retriever_searcher_args(self) -> dict[str, Any]:
+ """Searcher constructor args for the Retriever."""
+ index_dir = Path(self._assets_dir) / "indexes"
+ if self.searcher_type == "bm25":
+ return {"index_path": str(index_dir / "bm25")}
+ model_dir = self.searcher_model_name.lower().split("/")[-1]
+ return {
+ "index_path": str(index_dir / model_dir / "corpus.shard*_of_4.pkl"),
+ "model_name": self.searcher_model_name,
+ "normalize": self.normalize_search,
+ }
+
+ def _ensure_retriever(self) -> str:
+ """Start a shared retriever service and return its URL."""
+ assert self.retriever_runner is not None
+ proxy = get_shared_retriever(
+ runner=self.retriever_runner,
+ runner_kwargs=self._retriever_runner_kwargs(),
+ searcher_type=self.searcher_type,
+ **self._get_retriever_searcher_args(),
+ )
+ url = get_retriever_url(proxy)
+ # Rewrite URL for Docker sessions so they can reach the host.
+ if self.resolve_runner() == "docker":
+ url = url.replace("127.0.0.1", "host.docker.internal")
+ return url
+
+ def runner_kwargs(self) -> dict[str, Any]:
+ kw = super().runner_kwargs()
+ kw["health_timeout"] = 120.0
+ if self.resolve_runner() == "docker":
+ # Mount host assets (indexes, data) into the container so they
+ # are not re-downloaded for every image build.
+ volumes = kw.get("volumes", {})
+ volumes[self._assets_dir] = self._assets_dir
+ kw["volumes"] = volumes
+ return kw
+
+ def _get_evaluator_kwargs(self) -> dict[str, Any]:
+ kwargs: dict[str, Any] = {
+ "subset": self.subset,
+ "searcher_type": self.searcher_type,
+ "searcher_model_name": self.searcher_model_name,
+ "max_snippet_length": self.max_snippet_length,
+ "top_k_docs": self.top_k_docs,
+ "include_get_document": self.include_get_document,
+ "normalize_search": self.normalize_search,
+ "full_doc_max_tokens": self.full_doc_max_tokens,
+ "max_interactions": self.max_interactions,
+ "inference_model": self.inference_model,
+ "eval_model_id": self.eval_model_id,
+ }
+ # Auto-use a shared retriever service for Docker so that session
+ # containers don't each load the heavy search index (OOM).
+ if not self.retriever_runner and self.resolve_runner() == "docker":
+ self.retriever_runner = "service"
+ if self.retriever_runner:
+ kwargs["retriever_url"] = self._ensure_retriever()
+ return kwargs
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/browsecomp_eval.py b/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/browsecomp_eval.py
new file mode 100644
index 00000000..bd67e3c5
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/browsecomp_eval.py
@@ -0,0 +1,156 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+import json
+from typing import Any
+
+import litellm
+from pydantic import BaseModel
+from scripts_evaluation.evaluate_with_openai import (
+ GRADER_TEMPLATE as GRADER_TEMPLATE_OPENAI,
+)
+from scripts_evaluation.evaluate_with_openai import (
+ compute_citation_metrics,
+ extract_citations_from_response,
+ parse_judge_response,
+)
+from search_agent.prompts import GRADER_TEMPLATE_QWEN
+
+from ...core.context import try_get_context
+from ...core.types import SessionScore
+from ...utils.settings import get_settings
+
+_settings = get_settings()
+
+
+class BrowseCompEvaluator(BaseModel):
+ eval_model_id: str
+ sampling_params: dict[str, Any] = {}
+ grader_template: str
+
+ def evaluate_response(
+ self,
+ agent_response,
+ instance,
+ retrieved_docids_set=None,
+ tool_call_counts=None,
+ ) -> SessionScore:
+ question = instance["query"]
+ correct_answer = instance["gold_answer"]
+ positives_for_query = instance["evidence_docs"]
+ cited_docids = []
+ confidence = None
+ retrieval_recall = None
+ extracted_final_answer = None
+ judge_textual_response = None
+
+ # compute retrieval recall
+ if retrieved_docids_set is not None:
+ retrieval_recall = len(retrieved_docids_set.intersection(set(positives_for_query))) / float(
+ len(positives_for_query)
+ )
+
+ if not agent_response: # run was halted without a final answer
+ is_successful = False
+ score = 0
+ is_complete = False
+ parse_error = False
+ judge_usage = None
+
+ else: # call judge
+ is_complete = True
+ prompt = self.create_judge_prompt(question, agent_response, correct_answer)
+ judge_response = litellm.completion(
+ model=self.eval_model_id,
+ messages=[{"role": "user", "content": prompt}],
+ max_tokens=self.max_output_tokens,
+ litellm_metadata={"context": try_get_context()},
+ **self.sampling_params,
+ )
+ choice = judge_response["choices"][0]
+ judge_textual_response = choice.get("message").content
+ judge_usage = judge_response.usage.copy()
+ answer_metrics = parse_judge_response(judge_textual_response)
+ parse_error = bool(answer_metrics["parse_error"])
+
+ if parse_error:
+ is_successful = False
+ score = 0
+ else:
+ is_successful = answer_metrics["correct"]
+ score = int(is_successful) if is_successful is not None else 0
+ confidence = (answer_metrics.get("confidence", 100),)
+ extracted_final_answer = answer_metrics.get("extracted_final_answer")
+ cited_docids = extract_citations_from_response(agent_response)
+
+ citation_metrics_positives = compute_citation_metrics(cited_docids, positives_for_query)
+ scores = {
+ "Accuracy": score,
+ "Retrieval_recall": retrieval_recall,
+ "Citation_metrics_positives": citation_metrics_positives,
+ "Confidence": confidence,
+ }
+ meta_data = {
+ "instance": instance.copy(),
+ "retrieved_docids": list(retrieved_docids_set) if retrieved_docids_set else [],
+ "response": agent_response,
+ "extracted_final_answer": extracted_final_answer,
+ "judge_model": self.eval_model_id,
+ "is_complete": is_complete,
+ "judge_parse_error": parse_error,
+ "tool_call_counts": tool_call_counts,
+ }
+ if parse_error:
+ meta_data["judge_raw_response"] = judge_textual_response
+ all_scores = SessionScore(
+ score=int(is_successful),
+ success=is_successful,
+ is_finished=is_complete,
+ session_metrics=scores,
+ session_metadata=meta_data,
+ )
+
+ return all_scores, judge_usage
+
+ def create_judge_prompt(self, question: str, response: str, correct_answer: str) -> str:
+ return self.grader_template.format(question=question, response=response, correct_answer=correct_answer)
+
+
+class BrowseCompEvaluatorOpenai(BrowseCompEvaluator):
+ max_output_tokens: int = 1024
+ grader_template: str = GRADER_TEMPLATE_OPENAI
+ eval_model_id: str = "openai/Azure/gpt-4.1"
+
+
+class BrowsecompEvaluatorQwen(BrowseCompEvaluator):
+ max_output_tokens: int = 4096
+ temperature: int = 0.7
+ top_p: float = 0.8
+ top_k: int = 20
+ eval_model_id: str = "Qwen/Qwen3-32B"
+
+ def model_post_init(self, __context) -> None:
+ self.grader_template = GRADER_TEMPLATE_QWEN
+ self.sampling_params = {
+ "top_p": self.top_p,
+ "temperature": self.temperature,
+ "top_k": self.top_k,
+ }
+
+
+if __name__ == "__main__":
+ e = BrowseCompEvaluatorOpenai() # eval_model_id="watsonx/openai/gpt-oss-120b")
+ instance = {
+ "gold_answer": "2015",
+ "query": "When did someone was born",
+ "evidence_docs": ["1", "4"],
+ }
+ response = {
+ "explanation": "as stated in docs [1] [5]",
+ "exact_answer": "1925",
+ "confidence": 0.8,
+ }
+ response = json.dumps(response)
+
+ r = e.evaluate_response(response, instance)
+ print(r[0])
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/make_light_dataset.py b/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/make_light_dataset.py
new file mode 100644
index 00000000..db01af1c
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/make_light_dataset.py
@@ -0,0 +1,25 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+import argparse
+import json
+
+
+def main():
+ p = argparse.ArgumentParser(description="Create a light JSONL with only docids.")
+ p.add_argument("--input", required=True, help="Path to full JSONL")
+ p.add_argument("--output", required=True, help="Path to write the light JSONL")
+ args = p.parse_args()
+
+ with open(args.input, encoding="utf-8") as fin, open(args.output, "w", encoding="utf-8") as fout:
+ for line in fin:
+ if line.strip():
+ obj = json.loads(line)
+ for k in ["gold_docs", "evidence_docs", "negative_docs"]:
+ obj[k] = [d.get("docid") for d in obj.get(k) if d.get("docid") is not None]
+
+ fout.write(json.dumps(obj, ensure_ascii=False) + "\n")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/requirements.txt b/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/requirements.txt
new file mode 100644
index 00000000..3fc1116e
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/requirements.txt
@@ -0,0 +1,2 @@
+# browsecomp-plus is installed by setup.sh rather than here because it pins
+# conflicting versions of fastmcp, mcp, and pydantic.
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/retriever.py b/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/retriever.py
new file mode 100644
index 00000000..69d4ff8e
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/retriever.py
@@ -0,0 +1,133 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+"""Shared retriever service for BrowseCompPlus search index.
+
+The Retriever loads the heavy search index once and serves queries.
+It can run via any runner (direct, service, docker, etc.), allowing
+a single index copy to be shared across all sessions.
+"""
+
+import argparse
+import json
+import threading
+from typing import Any
+
+
+class Retriever:
+ """Loads a search index and serves queries.
+
+ Designed to run via ``with_runner()`` in any runner.
+ """
+
+ def __init__(self, searcher_type: str, **searcher_args: Any) -> None:
+ # Import torch/safetensors before the searcher module to avoid a
+ # native-library initialisation conflict with FAISS that causes
+ # segfaults on Apple Silicon (faiss_searcher.py imports faiss
+ # before torch at module level).
+ import safetensors # noqa: F401
+ import torch # noqa: F401
+ from searcher.searchers import SearcherType
+
+ searcher_class = SearcherType.get_searcher_class(searcher_type)
+ parser = argparse.ArgumentParser()
+ searcher_class.parse_args(parser)
+
+ cli: list[str] = []
+ for key, value in searcher_args.items():
+ flag = f"--{key.replace('_', '-')}"
+ if isinstance(value, bool):
+ if value:
+ cli.append(flag)
+ else:
+ cli.extend([flag, str(value)])
+
+ args = parser.parse_args(cli)
+ self._searcher = searcher_class(args)
+
+ def search(self, query: str, k: int) -> list:
+ return self._searcher.search(query, k)
+
+ def get_document(self, docid: str) -> dict | None:
+ return self._searcher.get_document(docid)
+
+
+class RetrieverClient:
+ """Lazy HTTP client to a remote Retriever service.
+
+ Picklable — stores only the URL. Connects on first use.
+ This allows it to survive serialization into Docker containers.
+ """
+
+ def __init__(self, url: str) -> None:
+ self._url = url
+ self._proxy: Any = None
+
+ def _connect(self) -> None:
+ if self._proxy is None:
+ from ...adapters.runners.service import HTTPTransport
+ from ...adapters.runners.transport import ObjectProxy
+
+ self._proxy = ObjectProxy(HTTPTransport(self._url))
+
+ def search(self, query: str, k: int) -> list:
+ self._connect()
+ return self._proxy.search(query, k)
+
+ def get_document(self, docid: str) -> dict | None:
+ self._connect()
+ return self._proxy.get_document(docid)
+
+ def close(self) -> None:
+ if self._proxy is not None:
+ try:
+ self._proxy.close()
+ except Exception:
+ pass
+ self._proxy = None
+
+ def __getstate__(self) -> dict:
+ return {"url": self._url}
+
+ def __setstate__(self, state: dict) -> None:
+ self._url = state["url"]
+ self._proxy = None
+
+
+# ── Shared retriever cache ────────────────────────────────────────────
+
+_cache_lock = threading.Lock()
+_cache: dict[str, Any] = {}
+
+
+def get_shared_retriever(
+ runner: str,
+ runner_kwargs: dict[str, Any] | None = None,
+ **retriever_kwargs: Any,
+) -> Any:
+ """Get or create a shared Retriever running in the specified runner."""
+ from ...adapters.runners import with_runner
+
+ key = json.dumps(retriever_kwargs, sort_keys=True, default=str)
+ if key not in _cache:
+ with _cache_lock:
+ if key not in _cache:
+ _cache[key] = with_runner(
+ Retriever,
+ runner=runner,
+ **(runner_kwargs or {}),
+ **retriever_kwargs,
+ )
+ return _cache[key]
+
+
+def get_retriever_url(proxy: Any) -> str:
+ """Extract the HTTP URL from a retriever proxy."""
+ from ...adapters.runners.service import HTTPTransport
+
+ transport = object.__getattribute__(proxy, "_transport")
+ if isinstance(transport, HTTPTransport):
+ return transport._base_url
+ raise ValueError(
+ "Cannot extract URL from non-HTTP retriever. Use runner='service' or runner='docker' for the retriever."
+ )
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/search_service.py b/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/search_service.py
new file mode 100644
index 00000000..d5bfbbd1
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/search_service.py
@@ -0,0 +1,173 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+"""Thread-safe search service with semaphore for concurrency control."""
+
+import argparse
+import atexit
+import json
+import logging
+import os
+import threading
+import time
+
+import psutil
+
+
+class SearchService:
+ """Thread-safe singleton service for managing searcher instances."""
+
+ _instance = None
+ _instance_lock = threading.Lock()
+
+ def __new__(cls):
+ if cls._instance is None:
+ with cls._instance_lock:
+ if cls._instance is None:
+ cls._instance = super().__new__(cls)
+ cls._instance._searchers = {}
+ cls._instance._cache_lock = threading.RLock()
+ cls._instance._search_semaphore = threading.Semaphore(5)
+ cls._instance._shutdown = False
+ # Register cleanup to run when program exits
+ atexit.register(cls._instance.shutdown)
+ return cls._instance
+
+ def shutdown(self):
+ """Clean up all searcher instances and their resources."""
+ with self._cache_lock:
+ if self._shutdown:
+ return
+ self._shutdown = True
+
+ logger = logging.getLogger(__name__)
+ logger.info("Shutting down SearchService and cleaning up models...")
+
+ for cache_key, searcher in self._searchers.items():
+ try:
+ # Try to close/cleanup the searcher if it has such methods
+ if hasattr(searcher, "close"):
+ searcher.close()
+ elif hasattr(searcher, "shutdown"):
+ searcher.shutdown()
+
+ if hasattr(searcher, "searcher") and hasattr(searcher.searcher, "close"):
+ searcher.searcher.close()
+
+ except Exception as e:
+ logger.warning(f"Error cleaning up searcher {cache_key}: {e}")
+
+ self._searchers.clear()
+ logger.info("SearchService shutdown complete")
+
+ def get_or_create_searcher(
+ self,
+ searcher_type: str,
+ searcher_class: type,
+ logger: logging.Logger | None = None,
+ **searcher_args,
+ ):
+ """Get or create a searcher instance with thread-safe access."""
+ if logger is None:
+ logger = logging.getLogger(__name__)
+
+ # Create cache key
+ args_str = json.dumps(searcher_args, sort_keys=True, default=str)
+ cache_key = f"{searcher_type}:{args_str}"
+
+ # Check if already cached
+ if cache_key in self._searchers:
+ logger.info(
+ f"Reusing cached searcher: {searcher_type} (PID: {os.getpid()}, Thread: {threading.get_ident()})"
+ )
+ return ThreadSafeSearcherWrapper(self._searchers[cache_key], self._search_semaphore)
+
+ # Create new searcher (with lock)
+ with self._cache_lock:
+ if cache_key in self._searchers:
+ return ThreadSafeSearcherWrapper(self._searchers[cache_key], self._search_semaphore)
+
+ process = psutil.Process(os.getpid())
+ mem_before = process.memory_info().rss / 1024 / 1024
+ thread_id = threading.get_ident()
+ logger.info(
+ f"Loading searcher model (before: {mem_before:.1f} MB, PID: {os.getpid()}, Thread: {thread_id})"
+ )
+
+ # Instantiate searcher
+ searcher = self._instantiate_with_overrides(searcher_class, **searcher_args)
+ self._searchers[cache_key] = searcher
+
+ mem_after = process.memory_info().rss / 1024 / 1024
+ logger.info(
+ f"Searcher model loaded: {searcher_type} "
+ f"(after: {mem_after:.1f} MB, delta: {mem_after - mem_before:.1f} MB)"
+ )
+
+ return ThreadSafeSearcherWrapper(searcher, self._search_semaphore)
+
+ @staticmethod
+ def _instantiate_with_overrides(cls, **overrides):
+ """Instantiate a class that uses argparse for configuration."""
+ parser = argparse.ArgumentParser()
+ cls.parse_args(parser)
+
+ cli = []
+ for key, value in overrides.items():
+ flag = f"--{key.replace('_', '-')}"
+ if isinstance(value, bool):
+ if value:
+ cli.append(flag)
+ else:
+ cli.extend([flag, str(value)])
+
+ args = parser.parse_args(cli)
+ return cls(args)
+
+
+class ThreadSafeSearcherWrapper:
+ """Wrapper that limits concurrent access to the searcher."""
+
+ def __init__(self, searcher, semaphore):
+ self._searcher = searcher
+ self._semaphore = semaphore
+ self._search_count = 0
+
+ def search(self, query: str, k: int):
+ """Thread-safe search with semaphore and lock for async safety."""
+ # Use both semaphore and lock to ensure serialization even in async contexts
+ acquired = self._semaphore.acquire(blocking=True, timeout=300)
+ if not acquired:
+ raise TimeoutError("Failed to acquire semaphore for search")
+ try:
+ self._search_count += 1
+ count = self._search_count
+
+ logger = logging.getLogger(__name__)
+ logger.debug(f"Search #{count} starting (thread={threading.get_ident()})")
+
+ result = self._searcher.search(query, k)
+ time.sleep(0.01)
+ logger.debug(f"Search #{count} completed")
+ return result
+ finally:
+ self._semaphore.release()
+
+ def get_document(self, docid: str):
+ """Thread-safe document retrieval with semaphore and lock."""
+ acquired = self._semaphore.acquire(blocking=True, timeout=300)
+ if not acquired:
+ raise TimeoutError("Failed to acquire semaphore for get_document")
+ try:
+ return self._searcher.get_document(docid)
+ finally:
+ self._semaphore.release()
+
+ def __getattr__(self, name):
+ """Forward other attributes to the wrapped searcher."""
+ return getattr(self._searcher, name)
+
+
+def get_search_service() -> SearchService:
+ """Get the singleton SearchService instance."""
+ return SearchService()
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/search_tool_handler.py b/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/search_tool_handler.py
new file mode 100644
index 00000000..d4ef5f27
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/search_tool_handler.py
@@ -0,0 +1,81 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+import json
+
+from transformers import AutoTokenizer
+
+
+class BCPSearchToolHandler:
+ def __init__(
+ self,
+ searcher,
+ snippet_max_tokens: int | None = None,
+ k: int = 5,
+ include_get_document: bool = True,
+ full_doc_max_tokens: int | None = None,
+ ):
+ self.searcher = searcher
+ self.snippet_max_tokens = snippet_max_tokens
+ self.k = k
+ self.include_get_document = include_get_document
+
+ self.tokenizer = None
+ self.full_doc_max_tokens = None
+ if snippet_max_tokens and snippet_max_tokens > 0:
+ self.tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
+ self.full_doc_max_tokens = full_doc_max_tokens
+
+ def execute_tool(self, tool_name: str, arguments: dict):
+ if tool_name == "search":
+ return self._search(arguments["query"])
+ if tool_name == "get_document":
+ return self._get_document(arguments["docid"])
+ raise ValueError(f"Unknown tool: {tool_name}")
+
+ def _search(self, query: str):
+ candidates = self.searcher.search(query, self.k)
+
+ if self.snippet_max_tokens and self.snippet_max_tokens > 0 and self.tokenizer:
+ for cand in candidates:
+ text = cand["text"]
+ cand["snippet"] = self._truncate_text(text, self.snippet_max_tokens)
+ else:
+ for cand in candidates:
+ cand["snippet"] = cand["text"]
+
+ results = []
+ for cand in candidates:
+ if cand.get("score") is None:
+ results.append({"docid": cand["docid"], "snippet": cand["snippet"]})
+ else:
+ results.append(
+ {
+ "docid": cand["docid"],
+ "score": cand["score"],
+ "snippet": cand["snippet"],
+ }
+ )
+
+ return json.dumps(results, indent=2)
+
+ def _get_document(self, docid: str):
+ try:
+ result = self.searcher.get_document(docid)
+ except Exception:
+ result = None
+ if result is None:
+ return json.dumps({"error": f"Document {docid} not found"})
+
+ text = result.get("text")
+ result["text"] = self._truncate_text(text, max_len=self.full_doc_max_tokens)
+ return json.dumps(result, indent=2)
+
+ def _truncate_text(self, text: str, max_len: int) -> str:
+ if not self.tokenizer:
+ raise RuntimeError("Tokenizer not initialized")
+ tokens = self.tokenizer.encode(text, add_special_tokens=False)
+ if max_len and len(tokens) > max_len:
+ truncated_tokens = tokens[:max_len]
+ return self.tokenizer.decode(truncated_tokens, skip_special_tokens=True)
+ return text
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/searcher_cache.py b/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/searcher_cache.py
new file mode 100644
index 00000000..86419cd3
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/searcher_cache.py
@@ -0,0 +1,69 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+import hashlib
+from typing import Any
+
+from ...utils.disk_cache import DiskCacheSessionMixin
+
+
+class SearchDiskCacheSession(DiskCacheSessionMixin):
+ CACHE_DIR = "./exgentic_session_cache/browsecomp/searcher" # single DB for all runs
+
+ def __init__(
+ self,
+ query: str,
+ n: int,
+ k: int,
+ search_type: str,
+ search_model: str,
+ normalize: bool = False,
+ use_cache: bool = True,
+ ):
+ self.query = query
+ self.n = n
+ self.k = k
+ self.search_type = search_type
+ self.search_model = search_model
+ self.normalize = normalize
+ self._init_cache_mixin(use_cache=use_cache)
+
+ def build_cache_key_payload(self) -> dict:
+ return {
+ "search_type": self.search_type,
+ "search_model": self.search_model,
+ "normalize": self.normalize,
+ "n": self.n,
+ "k": self.k,
+ "q": hashlib.sha256(self.query.encode("utf-8")).hexdigest(),
+ }
+
+ def build_additional_cache_metadata(self) -> dict:
+ return {
+ "search_type": self.search_type,
+ "search_model": self.search_model,
+ "normalize": self.normalize,
+ }
+
+ def on_cache_hit(self, payload: dict[str, Any]) -> bool:
+ results = payload.get("results")
+ if not isinstance(results, dict):
+ return False
+ result = results.get("raw")
+ if result is None or not isinstance(result, str):
+ return False
+ return True
+
+ def cache_results(self, result: str):
+ self.set_results_payload({"raw": result})
+ self.cache_score({})
+
+ def handle_start_fetch_results(self):
+ if self.handle_cache_start():
+ cache_results = self.get_results_payload()
+ if cache_results:
+ try:
+ return cache_results["raw"]
+ except Exception:
+ return None
+ return None
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/setup.sh b/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/setup.sh
new file mode 100644
index 00000000..3e902651
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/browsecompplus/setup.sh
@@ -0,0 +1,93 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+###############################################################################
+# 1. Detect java in PATH
+###############################################################################
+BM25_AVAILABLE=false
+if command -v java >/dev/null 2>&1; then
+ JAVA_VERSION_RAW="$(java -version 2>&1 | head -n1)" || true
+ JAVA_MAJOR="$(echo "$JAVA_VERSION_RAW" | sed -E 's/.*"([0-9]+).*/\1/')"
+ if [[ "$JAVA_MAJOR" =~ ^[0-9]+$ ]] && [ "$JAVA_MAJOR" -ge 21 ]; then
+ BM25_AVAILABLE=true
+ else
+ echo "[WARNING] Java 21+ not detected. BM25 searcher will not be available."
+ fi
+else
+ echo "[WARNING] Java not found. BM25 searcher will not be available."
+fi
+
+BENCH_ROOT="."
+
+###############################################################################
+# 2. Detect GPU availability
+###############################################################################
+GPU_AVAILABLE=false
+if command -v nvidia-smi >/dev/null 2>&1 && nvidia-smi >/dev/null 2>&1; then
+ GPU_AVAILABLE=true
+fi
+
+###############################################################################
+# 3. Install BrowseCompPlus packages
+###############################################################################
+GIT_SSH_URL="https://github.com/lilacheden/BrowseComp-Plus/"
+GIT_REF="mac-support-and-packaging"
+
+if [ "$GPU_AVAILABLE" = true ]; then
+ uv pip install "git+${GIT_SSH_URL}@${GIT_REF}#egg=browsecomp-plus[gpu]"
+else
+ uv pip install "git+${GIT_SSH_URL}@${GIT_REF}"
+fi
+
+uv pip install --upgrade "mcp>=1.24" "transformers>=4.53.2,<5.0" \
+ "pillow>=12.1.1" "fastmcp>=2.14.0" "fastapi-sso>=0.19.0" "openai>=2.9.0"
+uv pip uninstall gradio 2>/dev/null || true
+
+if [ "$GPU_AVAILABLE" = true ]; then
+ uv pip install --no-build-isolation flash-attn
+fi
+
+# In Docker builds, skip data/index downloads — they'll be mounted as volumes.
+if [ "${EXGENTIC_DOCKER_BUILD:-}" = "1" ]; then
+ echo "Docker build: skipping data and index downloads (will be mounted at runtime)"
+ exit 0
+fi
+
+###############################################################################
+# 4. Download + decrypt dataset
+###############################################################################
+DATA_DIR="${BENCH_ROOT}/data"
+QUERIES_DIR="${BENCH_ROOT}/topics-qrels"
+mkdir -p "${DATA_DIR}" "${QUERIES_DIR}"
+
+if [ -f "${DATA_DIR}/browsecomp_plus_decrypted.jsonl" ] && [ -f "${QUERIES_DIR}/queries.tsv" ]; then
+ echo "Dataset already exists, skipping download."
+else
+ python -m scripts_build_index.decrypt_dataset \
+ --output "${DATA_DIR}/browsecomp_plus_decrypted.jsonl" \
+ --generate-tsv "${QUERIES_DIR}/queries.tsv"
+fi
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+LIGHT_JSONL="${DATA_DIR}/browsecomp_plus_decrypted_docids.jsonl"
+if [ ! -f "${LIGHT_JSONL}" ]; then
+ PYTHONPATH="${SCRIPT_DIR}:${PYTHONPATH:-}" python -m make_light_dataset \
+ --input "${DATA_DIR}/browsecomp_plus_decrypted.jsonl" \
+ --output "${LIGHT_JSONL}"
+fi
+
+###############################################################################
+# 5. Download indexes
+###############################################################################
+uv pip install -U hf_transfer 2>/dev/null || true
+
+mkdir -p "${BENCH_ROOT}/indexes"
+cd "${BENCH_ROOT}/indexes"
+
+if [ "$BM25_AVAILABLE" = true ]; then
+ HF_HUB_ENABLE_HF_TRANSFER=1 hf download Tevatron/browsecomp-plus-indexes --repo-type=dataset --include="bm25/*" --local-dir .
+fi
+
+HF_HUB_ENABLE_HF_TRANSFER=1 hf download Tevatron/browsecomp-plus-indexes --repo-type=dataset --include="qwen3-embedding-8b/*" --local-dir .
+
+echo "BrowseCompPlus setup complete"
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/gsm8k/__init__.py b/labs/AgentStream/exgentic/src/exgentic/benchmarks/gsm8k/__init__.py
new file mode 100644
index 00000000..367fec41
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/gsm8k/__init__.py
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/gsm8k/gsm8k_benchmark.py b/labs/AgentStream/exgentic/src/exgentic/benchmarks/gsm8k/gsm8k_benchmark.py
new file mode 100644
index 00000000..868e4798
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/gsm8k/gsm8k_benchmark.py
@@ -0,0 +1,351 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from __future__ import annotations
+
+import ast
+import json
+import logging
+import operator as op
+import re
+from typing import Any, ClassVar, Literal
+
+from pydantic import (
+ BaseModel,
+ ConfigDict,
+ Field,
+ field_validator,
+)
+
+from ...core.actions import ActionsHandler, extract_argument
+from ...core.benchmark import Benchmark
+from ...core.evaluator import Evaluator
+from ...core.session import Session
+from ...core.types import (
+ Action,
+ ActionType,
+ BenchmarkResults,
+ EmptyObservation,
+ FinishAction,
+ Observation,
+ SessionIndex,
+ SessionScore,
+ SingleAction,
+ SingleObservation,
+)
+from ...observers.logging import get_logger
+from ...utils.paths import get_run_paths
+from ...utils.settings import ExgenticSettings, RunnerName, get_settings
+
+GSM8K_TOTAL_TASKS = 1319
+
+_run_logger: logging.Logger | None = None
+
+
+def _get_run_logger() -> logging.Logger:
+ """Benchmark-level logger that writes into the run's run log."""
+ global _run_logger
+ if _run_logger is None:
+ log_path = get_run_paths().tracker
+ _run_logger = get_logger(__name__, str(log_path))
+ return _run_logger
+
+
+def _parse_int(s: str | None) -> int | None:
+ if s is None:
+ return None
+ s = str(s).strip()
+ # allow "42\n" etc.
+ if re.fullmatch(r"[+-]?\d+", s):
+ return int(s)
+ return None
+
+
+_ALLOWED_BINOPS = {
+ ast.Add: op.add,
+ ast.Sub: op.sub,
+ ast.Mult: op.mul,
+ ast.Div: op.truediv,
+}
+_ALLOWED_UNARYOPS = {ast.UAdd: op.pos, ast.USub: op.neg}
+_ALLOWED_DESC = "numbers (ints/decimals), + - * /, parentheses, unary +/-. No names, functions, **, %, comparisons."
+
+
+def safe_evaluate(expression: str):
+ expr = (expression or "").strip()
+ if not expr:
+ return f"Invalid expression: empty. Allowed: {_ALLOWED_DESC} Got: {expression!r}"
+ try:
+ tree = ast.parse(expr, mode="eval")
+ except SyntaxError:
+ return f"Invalid syntax. Allowed: {_ALLOWED_DESC} Got: {expression!r}"
+
+ def ev(n):
+ if isinstance(n, ast.Expression):
+ return ev(n.body)
+ if isinstance(n, ast.Constant) and isinstance(n.value, (int, float)):
+ return n.value
+ if isinstance(n, ast.UnaryOp) and type(n.op) in _ALLOWED_UNARYOPS:
+ return _ALLOWED_UNARYOPS[type(n.op)](ev(n.operand))
+ if isinstance(n, ast.BinOp) and type(n.op) in _ALLOWED_BINOPS:
+ if isinstance(n.op, ast.Div) and ev(n.right) == 0:
+ return f"Invalid op: division by zero. Got: {expression!r}"
+ return _ALLOWED_BINOPS[type(n.op)](ev(n.left), ev(n.right))
+ return f"Invalid element: {type(n).__name__}. Allowed: {_ALLOWED_DESC} Got: {expression!r}"
+
+ try:
+ out = ev(tree)
+ return (
+ out
+ if isinstance(out, (int, float, str))
+ else f"Did not evaluate to a number. Allowed: {_ALLOWED_DESC} Got: {expression!r}"
+ )
+ except Exception as e:
+ return f"Error evaluating. Allowed: {_ALLOWED_DESC} Got: {expression!r}. Error: {type(e).__name__}: {e}"
+
+
+class GSM8kCalculateExpressionArgs(BaseModel):
+ expression: str = Field(..., description="Arithmetic expression using + - * / and parentheses.")
+
+
+class GSM8kCalculateExpressionAction(SingleAction):
+ name: Literal["calculate_expression"] = "calculate_expression"
+ arguments: GSM8kCalculateExpressionArgs
+
+
+class GSM8kFinishArgs(BaseModel):
+ answer: str | int = Field(..., description="Final answer as a single integer (string or int).")
+
+ @field_validator("answer", mode="before")
+ @classmethod
+ def coerce_int_to_str(cls, v: Any) -> str:
+ # Allow agents to pass raw integers; store as string for downstream checks.
+ if isinstance(v, int):
+ return str(v)
+ return v
+
+ @field_validator("answer")
+ @classmethod
+ def must_look_like_int(cls, v: str) -> str:
+ v = v.strip()
+ if not re.fullmatch(r"[+-]?\d+", v):
+ raise ValueError("Answer must be a single integer.")
+ return v
+
+
+class GSM8kFinishAction(FinishAction):
+ name: Literal["submit"] = "submit"
+ arguments: GSM8kFinishArgs
+
+
+class GSM8kSession(Session):
+ """Session for GSM8k benchmark evaluation."""
+
+ _question: str
+ _done: bool
+
+ def __init__(
+ self,
+ settings: ExgenticSettings,
+ include_calculator_tool: bool,
+ instance: dict[str, Any],
+ session_id: str | None = None,
+ ) -> None:
+ if session_id is not None:
+ self._session_id = session_id
+ self._question = instance["question"]
+ self._answer = instance["answer"]
+ self._task_id = instance["task_id"]
+ self._done = False
+ self._gold_answer = self._answer.split("####")[-1].strip()
+ self._final_answer = None
+ self._registry = ActionsHandler(logger=self.logger)
+ # Define Actions directly (single source of truth)
+
+ if include_calculator_tool:
+ self._registry.add_action(
+ name="calculate_expression",
+ description=(
+ "Evaluate a mathematical expression using only"
+ " numbers and basic operators"
+ " (+, -, *, /, parentheses)."
+ ),
+ action_cls=GSM8kCalculateExpressionAction,
+ handler=self._handle_calculate_expression,
+ )
+
+ self._registry.add_action(
+ name="submit",
+ description="Submit final answer and complete the task.",
+ action_cls=GSM8kFinishAction,
+ handler=self._handle_finish,
+ is_finish=True,
+ )
+ super().__init__()
+
+ @property
+ def task(self) -> str:
+ return (
+ "Solve the following math word problem using basic arithmetic.\n"
+ "You may perform intermediate calculations if helpful.\n"
+ "When you are finished, submit the final answer as a single integer by calling `submit`.\n"
+ "\n"
+ "Do not include units, words, or explanations in the final answer.\n"
+ "Your response will be graded only on whether the final integer exactly matches the correct answer.\n"
+ "\n"
+ f"Question:\n\n{self._question}"
+ )
+
+ @property
+ def context(self) -> dict[str, Any]:
+ return {}
+
+ @property
+ def actions(self) -> list[ActionType]:
+ return self._registry.actions
+
+ @property
+ def task_id(self) -> str:
+ return str(self._task_id)
+
+ def _to_observation(self, raw: Any, invoking_actions: list[SingleAction] | None = None) -> Observation:
+ return SingleObservation(invoking_actions=invoking_actions or [], result=raw)
+
+ def start(self) -> Observation | None:
+ # Empty initial observation; question is carried in the task string.
+ return EmptyObservation()
+
+ def step(self, action: Action) -> Observation | None:
+ if action is None:
+ self._done = True
+
+ if self._done:
+ return None
+
+ observation = self._registry.execute(action)
+
+ return observation
+
+ def done(self) -> bool:
+ return self._done
+
+ def score(self) -> SessionScore:
+ gold = _parse_int(self._gold_answer)
+ pred = _parse_int(self._final_answer)
+ score = 1.0 if (gold is not None and pred is not None and gold == pred) else 0.0
+ self.logger.info(f"Gold: {self._gold_answer} Prediction: {self._final_answer} Score: {score}")
+ # Finished only when the benchmark finish action stores a final answer.
+ finished = self._final_answer is not None
+ success = score == 1.0
+ return SessionScore(score=float(score), success=success, is_finished=finished)
+
+ def close(self):
+ super().close()
+ # Persist minimal results for aggregation
+ sc = self.score()
+ self.save_standard_results(sc)
+
+ # Action handlers ------------------------------------------------------------
+ def _handle_calculate_expression(self, action: SingleAction) -> Any:
+ self.logger.info(f"Received expression: {action}")
+ expression = extract_argument(action.arguments, "expression", "")
+ result = safe_evaluate(expression)
+ self.logger.info(f"Calculated result: {result}")
+ return result
+
+ def _handle_finish(self, action: SingleAction) -> None:
+ self.logger.info(f"Received final answer: {action}")
+ answer = extract_argument(action.arguments, "answer", None)
+ self._final_answer = answer
+ self._done = True
+ return
+
+
+# ── Evaluator ────────────────────────────────────────────────────────
+
+
+class GSM8kEvaluator(Evaluator):
+ """Evaluator for GSM8k — task discovery, session kwargs, aggregation."""
+
+ def __init__(self, subset: str = "main", include_calculator_tool: bool = True) -> None:
+ self._subset = subset
+ self._include_calculator_tool = include_calculator_tool
+ self._dataset = None
+
+ def _ensure_dataset(self) -> None:
+ if self._dataset is None:
+ from datasets import load_dataset
+
+ self._dataset = load_dataset("gsm8k", "main")["test"]
+
+ def list_tasks(self) -> list[str]:
+ return [str(i) for i in range(GSM8K_TOTAL_TASKS)]
+
+ def get_session_kwargs(self, index: SessionIndex) -> dict[str, Any]:
+ self._ensure_dataset()
+ idx = int(index.task_id)
+ if idx < 0 or idx >= len(self._dataset):
+ raise IndexError(f"Task id {index.task_id} out of range for GSM8k.")
+ instance = {"task_id": idx, **self._dataset[idx]}
+ return {
+ "settings": get_settings(),
+ "include_calculator_tool": self._include_calculator_tool,
+ "instance": instance,
+ "session_id": index.session_id,
+ }
+
+ def aggregate_sessions(self, sessions: list[SessionIndex]) -> BenchmarkResults:
+ run_logger = _get_run_logger()
+ scores: list[float] = []
+ for paths in self.get_sessions_paths(sessions):
+ fp = paths.benchmark_results
+ try:
+ with open(fp, encoding="utf-8-sig") as f:
+ payload = json.load(f)
+ s = float(payload["score"])
+ scores.append(s)
+ except FileNotFoundError as err:
+ raise FileNotFoundError(
+ f"Missing benchmark result for session" f" '{paths.session_id}' at {fp}"
+ ) from err
+ except Exception:
+ run_logger.exception(
+ "Failed to load benchmark result for session %s at %s",
+ paths.session_id,
+ fp,
+ )
+ raise
+ avg = sum(scores) / len(scores) if scores else 0.0
+ return BenchmarkResults(
+ benchmark_name="gsm8k",
+ total_tasks=len(sessions),
+ score=avg,
+ metrics={},
+ )
+
+
+# ── Benchmark config ─────────────────────────────────────────────────
+
+
+class GSM8kBenchmark(Benchmark, BaseModel):
+ display_name: ClassVar[str] = "GSM8k"
+ slug_name: ClassVar[str] = "gsm8k"
+ model_config = ConfigDict(arbitrary_types_allowed=True)
+
+ @classmethod
+ def _get_evaluator_class(cls):
+ return GSM8kEvaluator
+
+ @classmethod
+ def _get_session_class(cls):
+ return GSM8kSession
+
+ subset: Literal["main"] = "main"
+ include_calculator_tool: bool = True
+ runner: RunnerName | None = None # Threadsafe; uses global default runner
+
+ def _get_evaluator_kwargs(self) -> dict[str, Any]:
+ return {
+ "subset": self.subset,
+ "include_calculator_tool": self.include_calculator_tool,
+ }
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/gsm8k/requirements.txt b/labs/AgentStream/exgentic/src/exgentic/benchmarks/gsm8k/requirements.txt
new file mode 100644
index 00000000..aee11b28
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/gsm8k/requirements.txt
@@ -0,0 +1 @@
+datasets
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/hle/__init__.py b/labs/AgentStream/exgentic/src/exgentic/benchmarks/hle/__init__.py
new file mode 100644
index 00000000..367fec41
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/hle/__init__.py
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/hle/hle_benchmark.py b/labs/AgentStream/exgentic/src/exgentic/benchmarks/hle/hle_benchmark.py
new file mode 100644
index 00000000..d128f6b3
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/hle/hle_benchmark.py
@@ -0,0 +1,390 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from __future__ import annotations
+
+import asyncio
+import json
+import math
+from typing import Any, ClassVar, Literal
+
+import numpy as np
+from pydantic import BaseModel, ConfigDict, Field
+
+from ...core.actions import ActionsHandler, extract_argument
+from ...core.benchmark import Benchmark
+from ...core.evaluator import Evaluator
+from ...core.session import Session
+from ...core.types import (
+ Action,
+ ActionType,
+ BenchmarkResults,
+ EmptyObservation,
+ FinishAction,
+ Observation,
+ SessionIndex,
+ SessionScore,
+ SingleAction,
+ SingleObservation,
+)
+from ...utils.cost import CostReport, LiteLLMCostReport
+from ...utils.settings import RunnerName
+
+HLE_TOTAL_TASKS = 2500
+
+JUDGE_PROMPT = """Judge whether the following [response] to [question] is correct or not based on the precise and unambiguous [correct_answer] below.
+
+[question]: {question}
+
+[response]: {response}
+
+Your judgement must be in the format and criteria specified below:
+
+extracted_final_answer: The final exact answer extracted from the [response]. Put the extracted answer as 'None' if there is no exact, final answer to extract from the response.
+
+[correct_answer]: {correct_answer}
+
+reasoning: Explain why the extracted_final_answer is correct or incorrect based on [correct_answer], focusing only on if there are meaningful differences between [correct_answer] and the extracted_final_answer. Do not comment on any background to the problem, do not attempt to solve the problem, do not argue for any answer different than [correct_answer], focus only on whether the answers match.
+
+correct: Answer 'yes' if extracted_final_answer matches the [correct_answer] given above, or is within a small margin of error for numerical problems. Answer 'no' otherwise, i.e. if there if there is any inconsistency, ambiguity, non-equivalency, or if the extracted answer is incorrect.
+
+
+confidence: The extracted confidence score between 0|%| and 100|%| from [response]. Put 100 if there is no confidence score available."""
+
+
+class ExtractedAnswer(BaseModel):
+ extracted_final_answer: str
+ reasoning: str
+ correct: Literal["yes", "no"]
+ confidence: int
+ strict: Literal[True]
+
+
+class HLEFinishArgs(BaseModel):
+ explanation: str = Field(..., description="Your explanation/reasoning for your answer.")
+ answer: str = Field(..., description="Your final answer to the question.")
+ confidence: int = Field(..., description="Your confidence score between 0 and 100.", ge=0, le=100)
+
+
+class HLEFinishAction(FinishAction):
+ name: Literal["finish"] = "finish"
+ arguments: HLEFinishArgs
+
+class HLESession(Session):
+
+ def __init__(
+ self,
+ task_idx: int,
+ judge_model: str = "o3-mini-2025-01-31",
+ session_id: str | None = None,
+ agent_timeout: int = 900,
+ **_kwargs: Any,
+ ) -> None:
+ if session_id is not None:
+ self._session_id = session_id
+ from datasets import load_dataset
+
+ dataset = load_dataset("cais/hle", split="test")
+ row = dataset[task_idx]
+
+ self._question = row["question"]
+ self._gold_answer = row["answer"]
+ self._image = self._encode_image(row.get("image", None))
+ self._task_id = task_idx
+ self._judge_model = judge_model
+ self._agent_timeout = agent_timeout
+ self._start_time: float | None = None
+ self._done = False
+ self._final_answer: str | None = None
+ self._final_explanation: str | None = None
+ self._final_confidence: int | None = None
+ self._judge_result: dict[str, Any] | None = None
+ self._judge_input_tokens = 0
+ self._judge_output_tokens = 0
+
+ self._registry = ActionsHandler(logger=self.logger)
+ self._registry.add_action(
+ name="finish",
+ description="Submit your final answer with explanation and confidence score.",
+ action_cls=HLEFinishAction,
+ handler=self._handle_finish,
+ is_finish=True,
+ )
+ super().__init__()
+
+ @staticmethod
+ def _encode_image(image) -> str | None:
+ if image is None:
+ return None
+ if isinstance(image, str):
+ return image if image else None
+
+ import base64
+ import io
+
+ buf = io.BytesIO()
+ image.save(buf, format="PNG")
+ b64 = base64.b64encode(buf.getvalue()).decode("utf-8")
+ return f"data:image/png;base64,{b64}"
+
+ @property
+ def task(self) -> str:
+ return (
+ "Answer the following question. Submit your answer by calling the 'finish' action "
+ "with your explanation, final answer, and confidence score (0-100).\n\n"
+ f"Question: {self._question}"
+ )
+
+ @property
+ def context(self) -> dict[str, Any]:
+ if self._image:
+ return {"image": {"type": "image_url", "data": self._image}}
+ return {}
+
+ @property
+ def task_id(self) -> str:
+ return str(self._task_id)
+
+ @property
+ def actions(self) -> list[ActionType]:
+ return self._registry.actions
+
+ def start(self) -> Observation | None:
+ import time
+ self._start_time = time.time()
+ return EmptyObservation()
+
+ def step(self, action: Action) -> Observation | None:
+ import time
+ if action is None:
+ self._done = True
+ if self._done:
+ return None
+ if self._start_time is not None:
+ elapsed = time.time() - self._start_time
+ if elapsed >= self._agent_timeout:
+ self._done = True
+ return SingleObservation(
+ result=f"[Agent timeout reached ({self._agent_timeout}s). Session ending.]"
+ )
+ observation = self._registry.execute(action)
+ return observation
+
+ def done(self) -> bool:
+ return self._done
+
+ def _run_judge(self) -> dict[str, Any] | None:
+ if self._final_answer is None:
+ return None
+
+ response_text = (
+ f"Explanation: {self._final_explanation or ''}\n"
+ f"Answer: {self._final_answer}\n"
+ f"Confidence: {self._final_confidence or 100}%"
+ )
+
+ prompt = JUDGE_PROMPT.format(
+ question=self._question,
+ correct_answer=self._gold_answer,
+ response=response_text,
+ )
+
+ async def _judge_async() -> dict[str, Any] | None:
+ import litellm
+
+ try:
+ resp = await litellm.acompletion(
+ model=self._judge_model,
+ max_tokens=4096,
+ messages=[{"role": "user", "content": prompt}],
+ response_format=ExtractedAnswer,
+ )
+ import json as _json
+
+ content = _json.loads(resp.choices[0].message.content)
+ usage = getattr(resp, "usage", None)
+ if usage is not None:
+ self._judge_input_tokens += int(getattr(usage, "prompt_tokens", 0) or 0)
+ self._judge_output_tokens += int(getattr(usage, "completion_tokens", 0) or 0)
+ return {
+ "correct_answer": self._gold_answer,
+ "model_answer": content["extracted_final_answer"],
+ "reasoning": content["reasoning"],
+ "correct": content["correct"],
+ "confidence": content["confidence"],
+ }
+ except Exception as e:
+ self.logger.warning(f"Judge failed: {e}")
+ return None
+
+ return asyncio.run(_judge_async())
+
+ def score(self) -> SessionScore:
+ if self._judge_result is None:
+ self._judge_result = self._run_judge()
+
+ if self._judge_result is not None:
+ correct = self._judge_result["correct"] == "yes"
+ score = 1.0 if correct else 0.0
+ else:
+ score = 0.0
+
+ finished = self._final_answer is not None
+ return SessionScore(
+ score=score,
+ success=score == 1.0,
+ is_finished=finished,
+ session_metrics={
+ "confidence": self._final_confidence,
+ "judge_result": self._judge_result,
+ },
+ )
+
+ def close(self):
+ super().close()
+ sc = self.score()
+ self.save_standard_results(sc)
+
+ def _handle_finish(self, action: SingleAction) -> None:
+ self._final_explanation = extract_argument(action.arguments, "explanation", None)
+ self._final_answer = extract_argument(action.arguments, "answer", None)
+ self._final_confidence = extract_argument(action.arguments, "confidence", 100)
+ self._done = True
+ return None
+
+ def get_cost(self) -> CostReport:
+ if self._judge_input_tokens == 0 and self._judge_output_tokens == 0:
+ return LiteLLMCostReport.initialize_empty(model_name=self._judge_model)
+ return LiteLLMCostReport.from_token_counts(
+ self._judge_model,
+ self._judge_input_tokens,
+ self._judge_output_tokens,
+ )
+
+
+
+def calib_err(confidence, correct, p="2", beta=100):
+ idxs = np.argsort(confidence)
+ confidence = confidence[idxs]
+ correct = correct[idxs]
+ bins = [[i * beta, (i + 1) * beta] for i in range(len(confidence) // beta)]
+ if not bins:
+ return 0.0
+ bins[-1] = [bins[-1][0], len(confidence)]
+
+ cerr = 0
+ total_examples = len(confidence)
+ for i in range(len(bins) - 1):
+ bin_confidence = confidence[bins[i][0] : bins[i][1]]
+ bin_correct = correct[bins[i][0] : bins[i][1]]
+ num_examples_in_bin = len(bin_confidence)
+
+ if num_examples_in_bin > 0:
+ difference = np.abs(np.nanmean(bin_confidence) - np.nanmean(bin_correct))
+ if p == "2":
+ cerr += num_examples_in_bin / total_examples * np.square(difference)
+ elif p == "1":
+ cerr += num_examples_in_bin / total_examples * difference
+ elif p in ("infty", "infinity", "max"):
+ cerr = np.maximum(cerr, difference)
+
+ if p == "2":
+ cerr = np.sqrt(cerr)
+
+ return float(cerr)
+
+
+class HLEEvaluator(Evaluator):
+
+ def __init__(self, subset: str = "test", judge_model: str = "o3-mini-2025-01-31", agent_timeout: int = 900) -> None:
+ self._subset = subset
+ self._judge_model = judge_model
+ self._agent_timeout = agent_timeout
+ self._dataset = None
+
+ def _ensure_dataset(self) -> None:
+ if self._dataset is None:
+ from datasets import load_dataset
+
+ self._dataset = load_dataset("cais/hle", split="test")
+
+ def list_tasks(self) -> list[str]:
+ self._ensure_dataset()
+ return [str(i) for i in range(len(self._dataset))]
+
+ def get_session_kwargs(self, index: SessionIndex) -> dict[str, Any]:
+ self._ensure_dataset()
+ idx = int(index.task_id)
+ if idx < 0 or idx >= len(self._dataset):
+ raise IndexError(f"Task id {index.task_id} out of range for HLE.")
+ return {
+ "task_idx": idx,
+ "judge_model": self._judge_model,
+ "agent_timeout": self._agent_timeout,
+ "session_id": index.session_id,
+ }
+
+ def aggregate_sessions(self, sessions: list[SessionIndex]) -> BenchmarkResults:
+ scores: list[float] = []
+ confidences: list[float] = []
+ corrects: list[float] = []
+
+ for paths in self.get_sessions_paths(sessions):
+ with open(paths.benchmark_results, encoding="utf-8-sig") as f:
+ payload = json.load(f)
+ s = float(payload["score"])
+ scores.append(s)
+ corrects.append(s)
+ metrics = payload.get("session_metrics", {})
+ conf = metrics.get("confidence")
+ if conf is not None:
+ confidences.append(float(conf) / 100.0)
+ else:
+ confidences.append(1.0)
+
+ n = len(scores)
+ accuracy = 100 * sum(scores) / n if n else 0.0
+ confidence_half_width = 1.96 * math.sqrt(accuracy * (100 - accuracy) / n) if n else 0.0
+
+ cal_err = 0.0
+ if n > 0:
+ cal_err = 100 * calib_err(
+ np.array(confidences), np.array(corrects), p="2", beta=100
+ )
+
+ return BenchmarkResults(
+ benchmark_name="hle",
+ total_tasks=n,
+ score=accuracy / 100.0,
+ metrics={
+ "accuracy_pct": round(accuracy, 2),
+ "confidence_interval": round(confidence_half_width, 2),
+ "calibration_error": round(cal_err, 2),
+ },
+ )
+
+
+class HLEBenchmark(Benchmark, BaseModel):
+ display_name: ClassVar[str] = "HLE"
+ slug_name: ClassVar[str] = "hle"
+ model_config = ConfigDict(arbitrary_types_allowed=True)
+
+ @classmethod
+ def _get_evaluator_class(cls):
+ return HLEEvaluator
+
+ @classmethod
+ def _get_session_class(cls):
+ return HLESession
+
+ subset: Literal["test"] = "test"
+ judge_model: str = "o3-mini-2025-01-31"
+ agent_timeout: int = 900
+ runner: RunnerName | None = None
+
+ def _get_evaluator_kwargs(self) -> dict[str, Any]:
+ return {
+ "subset": self.subset,
+ "judge_model": self.judge_model,
+ "agent_timeout": self.agent_timeout,
+ }
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/hle/requirements.txt b/labs/AgentStream/exgentic/src/exgentic/benchmarks/hle/requirements.txt
new file mode 100644
index 00000000..bf051a4f
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/hle/requirements.txt
@@ -0,0 +1,3 @@
+datasets
+numpy
+litellm
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/hotpotqa/__init__.py b/labs/AgentStream/exgentic/src/exgentic/benchmarks/hotpotqa/__init__.py
new file mode 100644
index 00000000..367fec41
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/hotpotqa/__init__.py
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/hotpotqa/hotpotqa_benchmark.py b/labs/AgentStream/exgentic/src/exgentic/benchmarks/hotpotqa/hotpotqa_benchmark.py
new file mode 100644
index 00000000..8cea899f
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/hotpotqa/hotpotqa_benchmark.py
@@ -0,0 +1,373 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
+
+from __future__ import annotations
+
+import asyncio
+import json
+import re
+import stat
+import string
+import textwrap
+import threading
+from collections import Counter
+from typing import Any, ClassVar, Literal
+
+from pydantic import BaseModel, ConfigDict
+
+from ...adapters.schemas.openai import (
+ mcp_tools_to_openai_tools,
+ openai_tools_to_action_types,
+)
+
+# Copied scoring functions from https://github.com/hotpotqa/hotpot/blob/master/hotpot_evaluate_v1.py
+from ...core.actions import ActionsHandler, extract_argument
+from ...core.benchmark import Benchmark
+from ...core.evaluator import Evaluator
+from ...core.session import Session
+from ...core.types import (
+ Action,
+ ActionType,
+ BenchmarkResults,
+ EmptyObservation,
+ FinishAction,
+ Observation,
+ SessionIndex,
+ SessionScore,
+ SingleAction,
+ SingleObservation,
+)
+from ...utils.settings import RunnerName
+
+HOTPOTQA_TOTAL_TASKS = 7405
+
+
+def normalize_answer(s):
+ def remove_articles(text):
+ return re.sub(r"\b(a|an|the)\b", " ", text)
+
+ def white_space_fix(text):
+ return " ".join(text.split())
+
+ def remove_punc(text):
+ exclude = set(string.punctuation)
+ return "".join(ch for ch in text if ch not in exclude)
+
+ def lower(text):
+ return text.lower()
+
+ return white_space_fix(remove_articles(remove_punc(lower(s))))
+
+
+def f1_score(prediction, ground_truth):
+ normalized_prediction = normalize_answer(prediction)
+ normalized_ground_truth = normalize_answer(ground_truth)
+
+ zero_metric = (0, 0, 0)
+
+ if normalized_prediction in ["yes", "no", "noanswer"] and normalized_prediction != normalized_ground_truth:
+ return zero_metric
+ if normalized_ground_truth in ["yes", "no", "noanswer"] and normalized_prediction != normalized_ground_truth:
+ return zero_metric
+
+ prediction_tokens = normalized_prediction.split()
+ ground_truth_tokens = normalized_ground_truth.split()
+ common = Counter(prediction_tokens) & Counter(ground_truth_tokens)
+ num_same = sum(common.values())
+ if num_same == 0:
+ return zero_metric
+ precision = 1.0 * num_same / len(prediction_tokens)
+ recall = 1.0 * num_same / len(ground_truth_tokens)
+ f1 = (2 * precision * recall) / (precision + recall)
+ return f1, precision, recall
+
+
+class HotpotFinishArgs(BaseModel):
+ answer: str
+
+
+class HotpotFinishAction(FinishAction):
+ name: Literal["finish"] = "finish"
+ arguments: HotpotFinishArgs
+
+
+class HotpotQASession(Session):
+ """Session for HotpotQA benchmark evaluation."""
+
+ _question: str
+ _done: bool
+
+ def __init__(
+ self,
+ with_search_tools: bool,
+ instance: dict[str, Any],
+ session_id: str | None = None,
+ **_kwargs: Any,
+ ) -> None:
+ if session_id is not None:
+ self._session_id = session_id
+ self._question = instance["question"]
+ self.logger.info(f"question: {self._question}")
+ self._gold_answer = instance["answer"]
+ self._task_id = instance["task_id"]
+ self._done = False
+ self._final_answer = None
+ self._with_search_tools = with_search_tools
+ self._registry = ActionsHandler(logger=self.logger)
+ self._mcp_ready = threading.Event()
+ self._mcp_error: BaseException | None = None
+
+ self.mcp_thread: threading.Thread | None = None
+ if self._with_search_tools:
+ self.mcp_thread = threading.Thread(target=self.run_wikipedia_server, daemon=True)
+ self.mcp_thread.start()
+ ready = self._mcp_ready.wait(timeout=60.0)
+ if not ready or self._mcp_error is not None:
+ err = self._mcp_error
+ raise RuntimeError(f"MCP initialization failed or timed out: {err}") from err
+ else:
+ # No search tools requested; skip MCP startup.
+ self._mcp_ready.set()
+ # Only 'finish' is provided as the completion action
+ self._registry.add_action(
+ name="finish",
+ description="Submit the final answer and complete the task.",
+ action_cls=HotpotFinishAction,
+ handler=self._handle_finish,
+ is_finish=True,
+ )
+ super().__init__()
+
+ def run_wikipedia_server(self):
+ asyncio.run(self.run_wikipedia_server_async())
+
+ def _mcp_client_config(self) -> dict[str, Any]:
+ """Return a FastMCP client config that logs wikipedia-mcp stderr to the session benchmark dir."""
+ log_path = self.paths.benchmark_dir / "wikipedia_mcp.log"
+ log_path.parent.mkdir(parents=True, exist_ok=True)
+
+ from ...core.context import context_env
+
+ ctx_env = context_env()
+ ctx_env_json = json.dumps(ctx_env)
+
+ # Generate a tiny Python wrapper to keep stdout for JSONRPC and send stderr to a file.
+ wrapper_path = self.paths.benchmark_dir / "wikipedia_mcp_wrapper.py"
+ wrapper_code = (
+ textwrap.dedent(
+ f"""
+ #!/usr/bin/env python3
+ import subprocess, sys, os, json
+
+ log = open({str(log_path)!r}, "ab", buffering=0)
+ os.environ.update(json.loads({ctx_env_json!r}))
+ proc = subprocess.Popen(
+ ["wikipedia-mcp", "--transport", "stdio"],
+ stderr=log,
+ )
+ proc.wait()
+ sys.exit(proc.returncode)
+ """
+ ).strip()
+ + "\n"
+ )
+ wrapper_path.write_text(wrapper_code, encoding="utf-8")
+ wrapper_path.chmod(wrapper_path.stat().st_mode | stat.S_IXUSR | stat.S_IXGRP | stat.S_IXOTH)
+
+ command = str(wrapper_path)
+ return {"mcpServers": {"wiki": {"command": command, "args": []}}}
+
+ async def run_wikipedia_server_async(self):
+ if not self._with_search_tools:
+ self._mcp_ready.set()
+ return
+ try:
+ from fastmcp import Client
+
+ config = self._mcp_client_config()
+ mcp_client = Client(config)
+
+ async with mcp_client:
+ tools = await mcp_client.list_tools()
+
+ openai_tools = mcp_tools_to_openai_tools(tools)
+ if self._with_search_tools:
+ self._registry.add_actions(
+ openai_tools_to_action_types(openai_tools),
+ self._handle_mcp_action,
+ )
+ except Exception as e:
+ self._mcp_error = e
+ self.logger.exception(f"Failed to initialize wikipedia-mcp: {e}")
+ raise
+ finally:
+ self._mcp_ready.set()
+
+ @property
+ def task(self) -> str:
+ return (
+ "Answer the user question. Submit the final answer as a short phrase by calling 'finish'. "
+ "If the question is yes/no, answer 'yes' or 'no'. Do not add explanations.\n\n"
+ f"Question: {self._question}"
+ )
+
+ @property
+ def context(self) -> dict[str, Any]:
+ return {}
+
+ @property
+ def task_id(self) -> str:
+ return str(self._task_id)
+
+ @property
+ def actions(self) -> list[ActionType]:
+ return self._registry.actions
+
+ def _to_observation(self, raw: Any, invoking: list[SingleAction] | None = None) -> Observation:
+ return SingleObservation(invoking_actions=invoking or [], result=raw)
+
+ def start(self) -> Observation | None:
+ # Empty initial observation; question is carried in the task string.
+ return EmptyObservation()
+
+ def run_mcp_command(self, name, arguments) -> Any:
+ return asyncio.run(self.run_mcp_command_async(name, arguments))
+
+ async def run_mcp_command_async(self, name, arguments) -> Any:
+ from fastmcp import Client
+
+ config = self._mcp_client_config()
+ mcp_client = Client(config)
+
+ async with mcp_client:
+ response = await mcp_client.call_tool(name=name, arguments=arguments.model_dump())
+ # print(response.structured_content)
+ return response.structured_content
+
+ def step(self, action: Action) -> Observation | None:
+ if action is None:
+ self._done = True
+
+ if self._done:
+ return None
+
+ observation = self._registry.execute(action)
+
+ return observation
+
+ def done(self) -> bool:
+ return self._done
+
+ def score(self) -> SessionScore:
+ # Minimal: compute F1 and always mark success
+ try:
+ f1, precision, recall = f1_score(self._gold_answer, self._final_answer)
+ score = float(f1)
+ except Exception:
+ score = 0.0
+ self.logger.info(f"Gold: {self._gold_answer} Prediction: {self._final_answer} Score: {score}")
+ # Finished only when the benchmark finish action stores a final answer.
+ finished = self._final_answer is not None
+ success = score >= 1.0 - 1e-6
+ return SessionScore(score=score, success=success, is_finished=finished)
+
+ def close(self):
+ super().close()
+ # Persist minimal results for aggregation
+ sc = self.score()
+ self.save_standard_results(sc)
+ self.logger.debug("Closing MCP server..")
+ if self.mcp_thread and self.mcp_thread.is_alive():
+ self.logger.debug("Waiting for MCP server to shut down.")
+ self.mcp_thread.join(timeout=60.0)
+ if self.mcp_thread.is_alive():
+ self.logger.warning("MCP server thread did shutdown cleanly, continuing anyway.")
+ else:
+ self.logger.debug("MCP server shutdown cleanly.")
+
+ # Action handlers ------------------------------------------------------------
+ def _handle_finish(self, action: SingleAction) -> Any:
+ self.logger.info(f"Received final answer: {action}")
+ answer = extract_argument(action.arguments, "answer", None)
+ self._final_answer = answer
+ self._done = True
+ return None
+
+ def _handle_mcp_action(self, action: SingleAction) -> Any:
+ result = self.run_mcp_command(action.name, action.arguments)
+ return result
+
+
+# ── Evaluator ────────────────────────────────────────────────────────
+
+
+class HotpotQAEvaluator(Evaluator):
+ """Evaluator for HotpotQA — task discovery, session kwargs, aggregation."""
+
+ def __init__(self, subset: str = "distractor", with_search_tools: bool = True) -> None:
+ self._subset = subset
+ self._with_search_tools = with_search_tools
+ self._dataset = None
+
+ def _ensure_dataset(self) -> None:
+ if self._dataset is None:
+ from datasets import load_dataset
+
+ self._dataset = load_dataset("hotpotqa/hotpot_qa", "distractor")["validation"]
+
+ def list_tasks(self) -> list[str]:
+ return [str(i) for i in range(HOTPOTQA_TOTAL_TASKS)]
+
+ def get_session_kwargs(self, index: SessionIndex) -> dict[str, Any]:
+ self._ensure_dataset()
+ idx = int(index.task_id)
+ if idx < 0 or idx >= len(self._dataset):
+ raise IndexError(f"Task id {index.task_id} out of range for HotpotQA.")
+ instance = {"task_id": idx, **self._dataset[idx]}
+ return {
+ "with_search_tools": self._with_search_tools,
+ "instance": instance,
+ "session_id": index.session_id,
+ }
+
+ def aggregate_sessions(self, sessions: list[SessionIndex]) -> BenchmarkResults:
+ scores: list[float] = []
+ for paths in self.get_sessions_paths(sessions):
+ with open(paths.benchmark_results, encoding="utf-8-sig") as f:
+ payload = json.load(f)
+ s = float(payload["score"])
+ scores.append(s)
+ avg = sum(scores) / len(scores) if scores else 0.0
+ return BenchmarkResults(
+ benchmark_name="hotpotqa",
+ total_tasks=len(sessions),
+ score=avg,
+ metrics={},
+ )
+
+
+# ── Benchmark config ─────────────────────────────────────────────────
+
+
+class HotpotQABenchmark(Benchmark, BaseModel):
+ display_name: ClassVar[str] = "HotpotQA"
+ slug_name: ClassVar[str] = "hotpotqa"
+ model_config = ConfigDict(arbitrary_types_allowed=True)
+
+ @classmethod
+ def _get_evaluator_class(cls):
+ return HotpotQAEvaluator
+
+ @classmethod
+ def _get_session_class(cls):
+ return HotpotQASession
+
+ subset: Literal["distractor"] = "distractor"
+ with_search_tools: bool = True
+ runner: RunnerName | None = None # Threadsafe; uses global default runner (venv)
+
+ def _get_evaluator_kwargs(self) -> dict[str, Any]:
+ return {
+ "subset": self.subset,
+ "with_search_tools": self.with_search_tools,
+ }
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/hotpotqa/requirements.txt b/labs/AgentStream/exgentic/src/exgentic/benchmarks/hotpotqa/requirements.txt
new file mode 100644
index 00000000..5e9dae03
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/hotpotqa/requirements.txt
@@ -0,0 +1,3 @@
+datasets
+fastmcp
+wikipedia-mcp
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/swebench/__init__.py b/labs/AgentStream/exgentic/src/exgentic/benchmarks/swebench/__init__.py
new file mode 100644
index 00000000..367fec41
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/swebench/__init__.py
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: Apache-2.0
+# Copyright (C) 2026, The Exgentic organization and its contributors.
diff --git a/labs/AgentStream/exgentic/src/exgentic/benchmarks/swebench/config.yaml b/labs/AgentStream/exgentic/src/exgentic/benchmarks/swebench/config.yaml
new file mode 100644
index 00000000..aac16d0b
--- /dev/null
+++ b/labs/AgentStream/exgentic/src/exgentic/benchmarks/swebench/config.yaml
@@ -0,0 +1,106 @@
+# SWE-bench Benchmark Configuration
+
+# Session settings
+session:
+ timeout: 1000 # Default timeout for bash commands (seconds)
+ environment_pull_timeout: 600 # Timeout for Docker container startup/pull (seconds)
+ observation_size_limit: 10000 # Max characters in observation output
+ max_interactions: 200 # Max actions per session (null for unlimited)
+ timeout_template: |
+ The last command {command} timed out and has been killed.
+ The output of the command was:
+
+
+
+
+> [!NOTE]
+> This directory contains a locally adapted snapshot of the upstream
+> [Exgentic](https://github.com/Exgentic/exgentic) project used by
+> [AgentStream](../README.md). AgentStream-specific changes to this copy are
+> maintained in Sico, and it is not published as a separate Exgentic package
+> from this repository. Report issues with this bundled copy in the
+> [Sico issue tracker](https://github.com/microsoft/Sico/issues); report issues
+> with the original project to the upstream repository.
+
+
+
+```bash
+exgentic dashboard
+```
+
+---
+
+## Output Structure
+
+Each run creates its own directory under `outputs/
+
+```bash
+# Discover
+exgentic list benchmarks
+exgentic list subsets --benchmark tau2
+exgentic list tasks --benchmark tau2 --subset retail --limit 5
+exgentic list agents
+exgentic install --benchmark tau2
+exgentic install --benchmark tau2 --docker
+exgentic install --benchmark tau2 --local
+exgentic uninstall --benchmark tau2
+
+# Run
+exgentic evaluate --benchmark tau2 --agent tool_calling --subset airline --num-tasks 10
+exgentic batch run --benchmark tau2 --agent tool_calling --subset airline --num-tasks 10
+
+# Inspect
+exgentic status --benchmark tau2 --agent tool_calling --subset airline --num-tasks 10
+exgentic preview --benchmark tau2 --agent tool_calling --subset airline --num-tasks 10
+exgentic results --benchmark tau2 --agent tool_calling --subset airline --num-tasks 10
+
+# Analyze
+exgentic compare --agents tool_calling openai --benchmark tau2
+
+# Explore
+exgentic dashboard
+```
+
+---
+
+## Advanced
+
+### Model Configuration
+
+```bash
+exgentic evaluate --benchmark tau2 --agent tool_calling --subset retail --num-tasks 2 \
+ --set agent.model.temperature=0.2
+```
+
+Supported fields: `temperature`, `top_p`, `max_tokens`, `reasoning_effort`, `num_retries`, `retry_after`, `retry_strategy`
+
+### Run Limits
+
+```bash
+exgentic evaluate --benchmark tau2 --agent tool_calling --subset retail --num-tasks 2 \
+ --max-steps 100 --max-actions 100
+```
+
+Sessions stop at either limit and record `limit_reached` status. Default: 100 for both.
+
+### HuggingFace
+
+Use HuggingFace models or run evaluations on HuggingFace Jobs. See [docs/huggingface.md](./docs/huggingface.md).
+
+---
+
+## How It Works
+
+To learn more about Exgentic's architecture and design, see our [arXiv paper](https://arxiv.org/abs/2602.22953).
+
+## Development
+
+For local development, editing, and contributing, see [DEVELOPMENT.md](./DEVELOPMENT.md).
+
+## Contributing
+
+We welcome issues and pull requests! See [CONTRIBUTING.md](./CONTRIBUTING.md) for guidelines.
+
+## Citing Exgentic
+
+```bibtex
+@misc{bandel2026generalagentevaluation,
+ title={General Agent Evaluation},
+ author={Elron Bandel and Asaf Yehudai and Lilach Eden and Yehoshua Sagron and Yotam Perlitz and Elad Venezian and Natalia Razinkov and Natan Ergas and Shlomit Shachor Ifergan and Segev Shlomov and Michal Jacovi and Leshem Choshen and Liat Ein-Dor and Yoav Katz and Michal Shmueli-Scheuer},
+ year={2026},
+ url={https://arxiv.org/abs/2602.22953},
+}
+```
+
+## License
+
+Apache License 2.0 — see [LICENSE](LICENSE).
+
+## Support
+
+For questions about this bundled copy,
+[open an issue](https://github.com/microsoft/Sico/issues) in Sico. For the
+original Exgentic project, use the
+[upstream issue tracker](https://github.com/Exgentic/exgentic/issues).
diff --git a/labs/AgentStream/exgentic/docs/README.md b/labs/AgentStream/exgentic/docs/README.md
new file mode 100644
index 00000000..c917d4f9
--- /dev/null
+++ b/labs/AgentStream/exgentic/docs/README.md
@@ -0,0 +1,38 @@
+# Exgentic Documentation
+
+Welcome to the Exgentic docs. Use the table below to find what you need.
+
+---
+
+## Using Exgentic
+
+| Document | Description |
+|----------|-------------|
+| [CLI Reference](./cli-reference.md) | Every command, flag, and environment variable |
+| [Python API](./python-api.md) | `evaluate()`, `execute()`, `aggregate()`, `status()`, `list_*()`, and all other library functions |
+| [Custom Models](./custom-models.md) | Use any LLM provider (OpenAI, Anthropic, Azure, Bedrock, Ollama, and more) via LiteLLM |
+| [Batch Runs](./batch.md) | Run parameter sweeps, manage large evaluations, export to CSV, publish to HuggingFace |
+| [Runners](./runners.md) | `direct`, `venv`, `docker` — isolation levels, configuration, Docker-in-Docker |
+| [Output Format](./output-format.md) | Schema for `results.json`, `trajectory.jsonl`, session results, and cost reports |
+| [Observers](./observers.md) | Hook into the evaluation lifecycle for custom logging, monitoring, and early stopping |
+
+## Extending Exgentic
+
+| Document | Description |
+|----------|-------------|
+| [Adding Agents](./adding-agents.md) | Write a new agent adapter — design principles, required methods, file layout, and validation checklist |
+| [Adding Benchmarks](./adding-benchmarks.md) | Write a new benchmark adapter — design principles, contract rules, and validation checklist |
+| [Replay Testing](./replay-testing.md) | Test benchmark and agent adapters end-to-end without API calls, using recorded sessions |
+
+## Observability
+
+| Document | Description |
+|----------|-------------|
+| [Quick Start](./observability/quickstart.md) | Set up OpenTelemetry tracing with Jaeger in five minutes |
+| [Semantic Conventions](./observability/semantic-conventions.md) | Full reference of every span and attribute Exgentic emits |
+
+## Other resources
+
+- [README.md](../README.md) — project overview, quick start, CLI reference, and available benchmarks/agents
+- [DEVELOPMENT.md](../DEVELOPMENT.md) — local setup, running tests, and linting
+- [CONTRIBUTING.md](../CONTRIBUTING.md) — contribution workflow, legal requirements, and PR guidelines
diff --git a/labs/AgentStream/exgentic/docs/adding-agents.md b/labs/AgentStream/exgentic/docs/adding-agents.md
new file mode 100644
index 00000000..2f6b3f2f
--- /dev/null
+++ b/labs/AgentStream/exgentic/docs/adding-agents.md
@@ -0,0 +1,384 @@
+# Adding Agents
+
+This document defines the agent design principles for Exgentic.
+
+It is intentionally opinionated. An agent adapter should not just "work"; it should cleanly separate configuration from execution, isolate heavy dependencies, and adapt to any benchmark contract without requiring the benchmark to change.
+
+Use these existing adapters as reference points:
+- `src/exgentic/agents/litellm_tool_calling/litellm_tool_calling_agent.py` + `instance.py` (split pattern)
+- `src/exgentic/agents/cli/claude/agent.py` (light pattern, single file)
+
+**Related docs:**
+[docs/](./README.md) · [Adding Benchmarks](./adding-benchmarks.md) · [Custom Models](./custom-models.md) · [Runners](./runners.md) · [Replay Testing](./replay-testing.md) · [CONTRIBUTING.md](../CONTRIBUTING.md)
+
+## Core Principle
+
+The agent adapts to the benchmark contract, not the other way around.
+
+That means:
+- the benchmark decides the task, context, actions, step flow, and scoring
+- the agent receives these through `_get_instance_kwargs()` and must work within them
+- the agent should not require benchmark modifications to function
+- the agent should not impose protocol-specific assumptions on the benchmark
+
+The default goal should be the thinnest possible agent wrapper.
+
+That means:
+- translate the benchmark's actions into whatever protocol your agent uses (tool calls, code generation, CLI commands)
+- do not reshape the benchmark contract to match your model's preferred format
+- keep the configuration surface small and explicit
+
+## Architecture
+
+Exgentic agents are split into two classes with distinct roles:
+
+### Agent (config, host-side)
+
+`Agent` is a lightweight Pydantic model that holds configuration. It lives on the host and is never sent into an isolated runner. It has no heavy dependencies.
+
+Responsibilities:
+- declare `display_name` and `slug_name` as `ClassVar[str]`
+- hold user-facing configuration fields (model name, max steps, feature flags)
+- implement `_get_instance_class()` to resolve the execution class
+- implement `_get_instance_kwargs()` to translate config + benchmark contract into constructor arguments
+- optionally override `setup()` for non-pip setup (Docker builds, npm installs)
+- optionally override `model_name` / `get_models_names()` for dashboard metadata
+
+### AgentInstance (execution, venv-side)
+
+`AgentInstance` is the execution class. It runs inside the runner (venv, Docker, or local) and may import heavy third-party libraries.
+
+Responsibilities:
+- implement `react(observation) -> Action | None` as the core decision loop
+- implement `close()` for resource cleanup
+- optionally override `start()` for initialization that happens after construction
+- optionally override `get_cost()` to report monetary cost
+
+The agent instance receives a single `session_id` in its constructor, which scopes all logs and artifacts. Additional kwargs come from `_get_instance_kwargs()`.
+
+## Key Pattern: Lazy Import for Dependency Isolation
+
+The `_get_instance_class()` classmethod must use a lazy import so that heavy dependencies are only loaded inside the runner environment, not on the host.
+
+```python
+@classmethod
+def _get_instance_class(cls):
+ from .instance import MyAgentInstance
+
+ return MyAgentInstance
+```
+
+This is the same pattern that `Benchmark._get_session_class()` uses. It ensures the host process never imports libraries like `litellm`, `smolagents`, `openai`, or any other agent-specific SDK.
+
+## When to Split Files
+
+**Split into separate files** when your agent depends on heavy third-party libraries:
+
+```
+src/exgentic/agents/my_agent/
+ __init__.py
+ my_agent.py # Agent subclass (light, no heavy imports)
+ instance.py # AgentInstance subclass (imports litellm, openai, etc.)
+ requirements.txt # Agent-specific pip dependencies
+ setup.sh # Optional non-pip setup
+ utils.py # Optional helpers
+```
+
+Examples: `litellm_tool_calling`, `smolagents`, `openai`
+
+**Keep everything in one file** when dependencies are light or already available in the base environment:
+
+```
+src/exgentic/agents/my_agent/
+ __init__.py
+ agent.py # Both Agent and AgentInstance in one file
+```
+
+Example: `cli/claude` (the instance class is in the same file because it only depends on stdlib and core Exgentic types)
+
+The rule is simple: if importing the instance class would pull in packages that are not in the base `exgentic` install, split the files.
+
+## Required Methods
+
+### On the Agent class
+
+#### `_get_instance_class()` (classmethod, abstract)
+
+Returns the `AgentInstance` subclass. Must use a lazy import.
+
+```python
+@classmethod
+def _get_instance_class(cls):
+ from .instance import MyAgentInstance
+
+ return MyAgentInstance
+```
+
+#### `_get_instance_kwargs()` (abstract)
+
+Translates the agent's configuration into constructor kwargs for the instance class. Task, context, and actions are passed separately via `start()`, not through the constructor.
+
+```python
+def _get_instance_kwargs(self, session_id: str) -> dict[str, Any]:
+ return {
+ "session_id": session_id,
+ "model": self.model,
+ "max_steps": self.max_steps,
+ }
+```
+
+The returned dict is passed directly to the instance class constructor. Every key must match a constructor parameter.
+
+### On the AgentInstance class
+
+#### `react(observation) -> Action | None` (abstract)
+
+The core decision loop. Receives an `Observation` (or `None` on the first call) and returns an `Action` to take, or `None` to signal that the agent is done.
+
+```python
+def react(self, observation: Observation | None) -> Action | None:
+ # Process observation, decide next action
+ # Return None when the agent decides it is finished
+ ...
+```
+
+#### `close()` (abstract)
+
+Cleanup resources. Called when the session ends, whether or not the agent finished normally.
+
+```python
+def close(self) -> None:
+ # Release connections, flush logs, etc.
+ pass
+```
+
+#### `start(task, context, actions)` (optional override)
+
+Called after construction but before the first `react()`. Receives the benchmark's task string, context dict, and list of action types. The base implementation stores these as `self.task`, `self.context`, and `self.actions`. Override to perform initialization that depends on these values (e.g., seeding a conversation with the task prompt).
+
+#### `get_cost()` (optional)
+
+Returns a `CostReport` with estimated monetary cost. Default returns an empty report. Override to track API costs.
+
+```python
+def get_cost(self) -> CostReport:
+ return self._cost_data
+```
+
+## Registration
+
+Every agent must be registered in `src/exgentic/interfaces/registry.py` in the `AGENTS` dict.
+
+```python
+AGENTS: dict[str, RegistryEntry] = {
+ # ...existing entries...
+ "my_agent": RegistryEntry(
+ slug_name="my_agent",
+ display_name="My Agent",
+ module="exgentic.agents.my_agent.my_agent",
+ attr="MyAgent",
+ kind="agent",
+ ),
+}
+```
+
+Requirements:
+- `slug_name` must match the `slug_name` ClassVar on the Agent class exactly
+- `display_name` must match the `display_name` ClassVar on the Agent class exactly
+- `module` is the dotted Python module path to the file containing the Agent class
+- `attr` is the class name within that module
+- `kind` must be `"agent"`
+
+The registry validates these constraints at load time. Mismatches will raise at startup.
+
+## Setup
+
+### `requirements.txt`
+
+List agent-specific pip dependencies. The runner installs these automatically into the isolated environment.
+
+```
+litellm>=1.50.0
+```
+
+Place the file in the agent's package directory. The `RunnerMixin` auto-discovers it by walking up from the module file.
+
+### `setup.sh`
+
+Optional script for non-pip setup. Runs after dependencies are installed.
+
+```bash
+#!/usr/bin/env bash
+set -euo pipefail
+# Build Docker images, install npm packages, download models, etc.
+```
+
+Place it next to the agent module. The `RunnerMixin` auto-discovers it.
+
+Both files are automatically found by the framework through `RunnerMixin.requirements_txt` and `RunnerMixin.setup_script`. No manual wiring is needed.
+
+## Recommended File Structure
+
+### Split pattern (heavy deps)
+
+```
+src/exgentic/agents/my_agent/
+ __init__.py
+ my_agent.py # Agent subclass
+ instance.py # AgentInstance subclass
+ requirements.txt # e.g., litellm>=1.50.0
+ setup.sh # optional
+ utils.py # optional helpers
+```
+
+**my_agent.py** (host-side, no heavy imports):
+
+```python
+from __future__ import annotations
+
+from typing import Any, ClassVar
+
+from ...core.agent import Agent
+from ...core.types import ActionType, ModelSettings
+
+
+class MyAgent(Agent):
+ display_name: ClassVar[str] = "My Agent"
+ slug_name: ClassVar[str] = "my_agent"
+
+ model: str = "gpt-4o"
+ max_steps: int = 100
+
+ @classmethod
+ def _get_instance_class(cls):
+ from .instance import MyAgentInstance
+
+ return MyAgentInstance
+
+ @property
+ def model_name(self) -> str:
+ return self.model
+
+ def _get_instance_kwargs(self, session_id: str) -> dict[str, Any]:
+ return {
+ "session_id": session_id,
+ "model": self.model,
+ "max_steps": self.max_steps,
+ }
+```
+
+**instance.py** (runner-side, may import heavy libs):
+
+```python
+from __future__ import annotations
+
+from typing import Any, Optional
+
+import some_heavy_library # only loaded inside the runner
+
+from ...core.agent_instance import AgentInstance
+from ...core.types import Action, ActionType, Observation
+
+
+class MyAgentInstance(AgentInstance):
+ def __init__(
+ self,
+ session_id: str,
+ task: str,
+ context: dict[str, Any],
+ actions: list[ActionType],
+ model: str,
+ max_steps: int,
+ ):
+ super().__init__(session_id)
+ self.task = task
+ self.context = context
+ self.actions = actions
+ self.model = model
+ self.max_steps = max_steps
+ self._step_count = 0
+
+ def react(self, observation: Optional[Observation]) -> Optional[Action]:
+ self._step_count += 1
+ if self._step_count > self.max_steps:
+ return None
+ # Agent decision logic here
+ ...
+
+ def close(self) -> None:
+ pass
+```
+
+### Light pattern (no heavy deps)
+
+```
+src/exgentic/agents/my_agent/
+ __init__.py
+ agent.py # Both Agent and AgentInstance
+```
+
+Keep both classes in a single file when the instance has no third-party imports beyond what exgentic already provides.
+
+## Validation Checklist
+
+Before opening a PR for a new agent, validate all of the following.
+
+### Contract validation
+
+- Agent class declares `display_name` and `slug_name` as `ClassVar[str]`
+- `_get_instance_class()` uses a lazy import
+- `_get_instance_kwargs()` returns a dict whose keys match the instance constructor
+- The instance implements `react()` and `close()`
+- The instance calls `super().__init__(session_id)` in its constructor
+
+### Dependency isolation validation
+
+- The Agent file does not import heavy third-party libraries at module level
+- Heavy imports only appear inside `_get_instance_class()` or in the instance module
+- `requirements.txt` lists all agent-specific dependencies
+
+### Registry validation
+
+- `slug_name` in the registry entry matches the class `slug_name` exactly
+- `display_name` in the registry entry matches the class `display_name` exactly
+- `module` path resolves to the correct file
+- `attr` matches the Agent class name
+- `kind` is `"agent"`
+
+### Functional validation
+
+- Agent is discoverable through the registry (`load_agent("my_agent")` succeeds)
+- Agent works with at least one benchmark end to end
+- `react()` correctly returns `None` when the agent decides it is done
+- `close()` does not raise
+- `get_cost()` returns a valid `CostReport`
+
+### Quality validation
+
+- `py_compile` passes for all changed Python files
+- `pre-commit` passes for changed files
+- `git diff --check` passes
+
+## Practical Rule of Thumb
+
+When in doubt, ask:
+
+1. Does the agent adapt to the benchmark, or does it require the benchmark to change?
+2. Are heavy dependencies isolated behind a lazy import?
+3. Is the Agent file importable without installing agent-specific packages?
+4. Does `_get_instance_kwargs()` faithfully pass the benchmark contract through?
+5. Will this agent work with benchmarks that have very different action spaces?
+6. Is the configuration surface minimal and explicit?
+
+If the answer to any of those is no, the adapter is probably too coupled or too leaky.
+
+---
+
+## See also
+
+- [Adding Benchmarks](./adding-benchmarks.md) — the other side of the contract
+- [Custom Models](./custom-models.md) — configuring LLM providers and sampling parameters for the `tool_calling` agent
+- [Runners](./runners.md) — how setup.sh and requirements.txt are discovered and executed
+- [Replay Testing](./replay-testing.md) — write end-to-end tests for your agent without API calls
+- [CONTRIBUTING.md](../CONTRIBUTING.md) — PR workflow and legal requirements
+- [docs/](./README.md) — documentation index
diff --git a/labs/AgentStream/exgentic/docs/adding-benchmarks.md b/labs/AgentStream/exgentic/docs/adding-benchmarks.md
new file mode 100644
index 00000000..4fd56287
--- /dev/null
+++ b/labs/AgentStream/exgentic/docs/adding-benchmarks.md
@@ -0,0 +1,364 @@
+# Adding Benchmarks
+
+This document defines the benchmark design principles for Exgentic.
+
+It is intentionally opinionated. A benchmark adapter should not just "work"; it should preserve the benchmark's meaning while still fitting Exgentic's agent abstraction cleanly.
+
+Use these existing adapters as reference points:
+- `src/exgentic/benchmarks/tau2/tau2_benchmark.py`
+- `src/exgentic/benchmarks/bfcl/bfcl_benchmark.py`
+
+**Related docs:**
+[docs/](./README.md) · [Adding Agents](./adding-agents.md) · [Runners](./runners.md) · [Replay Testing](./replay-testing.md) · [Output Format](./output-format.md) · [CONTRIBUTING.md](../CONTRIBUTING.md)
+
+## Core Principle
+
+The benchmark owns the contract.
+
+That means the benchmark decides:
+- what the task is
+- what context the agent receives
+- what actions exist
+- how steps progress
+- when a session is finished
+- how scoring works
+
+The agent should adapt to the benchmark contract through Exgentic's normal interfaces. The benchmark should not be shaped around one specific model protocol.
+
+The default goal should be the thinnest possible benchmark wrapper.
+
+That means:
+- reuse the source benchmark wherever possible
+- add only the translation layers that are actually necessary
+- avoid reimplementing benchmark logic unless there is a clear reason
+- avoid introducing runtime behavior that exists only to satisfy one agent or one model protocol
+
+The target is simple:
+- make the benchmark accessible to any Exgentic agent
+- while adding the minimum adapter surface necessary
+- and without clashing with agent-specific assumptions
+
+## Principles
+
+### 1. Keep the agent-facing contract protocol-agnostic
+
+Do not define a benchmark in terms of OpenAI tool calls, raw assistant messages, or any other provider-specific response format.
+
+Define it in terms of:
+- task semantics
+- available actions
+- observations
+- finish conditions
+- score
+
+Protocol-specific translation belongs in adapters, not in the benchmark contract.
+
+Bad:
+- "The model must return all tool calls in one assistant message."
+
+Good:
+- "The task is complete when the required actions have been taken and the benchmark-specific finish condition is met."
+
+### 2. The task should be the real task
+
+`task` should contain the actual task the agent is meant to solve.
+
+Do not wrap the task in fake chat scaffolding unless that scaffolding is genuinely part of the benchmark.
+
+If the benchmark is not about user interaction, do not invent a chat conversation just to make it look conversational.
+
+Bad:
+- synthetic "user" messages when the benchmark is not actually testing user interaction
+- generic wrapper prompts replacing the real benchmark task
+
+Good:
+- the benchmark prompt itself is the `task`
+
+### 3. Context should contain only what the agent should know
+
+`context` is not a metadata dump.
+
+It should contain only information that is necessary for the agent to behave correctly on the task.
+
+Keep internal benchmark metadata out of `context`, including:
+- subset names
+- source dataset ids
+- registry information
+- adapter implementation details
+
+Good context:
+- policy text
+- execution constraints
+- information the agent genuinely needs to act correctly
+
+Bad context:
+- `"subset": "live_parallel_multiple"`
+- `"benchmark": "bfcl"`
+
+### 4. Actions should represent semantic operations
+
+Actions are the benchmark's action space.
+
+Name and describe them in terms of what they do, not in terms of a transport protocol.
+
+Prefer "actions" over protocol-specific terms like "tool calls" in benchmark-facing language, because not all agents consume or produce actions through the same protocol.
+
+If the source benchmark exposes functions, commands, or tools, translate those into Exgentic actions at the boundary.
+
+### 5. Use `finish` only as part of the benchmark contract
+
+`finish` is valid when the benchmark needs an explicit end-of-step or end-of-task signal.
+
+It should exist because the benchmark contract needs it, not because a specific model API needs it.
+
+Use it when:
+- the benchmark has multiple steps or turns and needs an explicit transition point
+- the benchmark needs a clear "done with this step" signal
+- the benchmark should allow completion without another normal action
+
+Do not force `finish` into a benchmark if the source benchmark's semantics are cleaner without it.
+
+### 6. Distinguish execution modes by contract, not by protocol
+
+If a benchmark has single-turn, live, or multi-turn variants, define those as execution contracts.
+
+The important differences are things like:
+- whether more steps may follow
+- whether the task ends after the current finish
+- whether action outputs affect later state
+- whether the benchmark continues after a step completes
+
+Do not define the mode in terms of how many assistant messages or tool-call payloads a model is allowed to emit.
+
+Important:
+- single-turn does not necessarily mean a single action
+- multi-action single-turn tasks are valid
+- the distinction is about step structure, not about one specific model protocol
+
+### 7. Action outputs must be honest
+
+If the benchmark can produce real execution outputs, use them.
+
+If it cannot, do not fabricate realistic outputs that imply more runtime semantics than actually exist.
+
+Be explicit in the contract when actions are only being recorded rather than executed.
+
+Good:
+- real execution results when the source benchmark exposes an official executor
+- `Action recorded.` when there is no real runtime execution for that task family
+
+Bad:
+- made-up outputs that look like real environment state changes when none were actually computed
+
+### 8. Reuse external harnesses as the source of truth where possible
+
+When adapting an external benchmark, prefer to reuse:
+- dataset loading
+- official assets
+- ground-truth files
+- official checkers or scorers
+- official execution helpers
+
+Avoid copying large chunks of benchmark logic into Exgentic if the source repository already provides them.
+
+But there is an important boundary:
+- external harnesses should be the source of truth for benchmark assets and scoring
+- they should not automatically own the Exgentic runtime contract
+
+If the external harness assumes a model-specific interaction pattern, Exgentic should usually keep its own runtime and bridge to the harness at load/score time instead.
+
+When choosing between two valid integrations, prefer the thinner one.
+
+Use the more complex approach only when the thinner one would:
+- distort benchmark meaning
+- hard-code one agent's assumptions
+- or force Exgentic to own logic that should stay with the source benchmark
+
+### 9. Be explicit about what is official and what is adapted
+
+If the adapter preserves official scoring but changes runtime behavior, document that clearly.
+
+If some subsets use official execution while others only use official scoring, document that too.
+
+Do not imply full equivalence when the integration is intentionally more abstract than the source benchmark.
+
+For each benchmark adapter, it should be easy to answer:
+- What comes directly from the source benchmark?
+- What is adapted by Exgentic?
+- What is exact?
+- What is approximate?
+
+### 10. Success, failure, and error must stay distinct
+
+Finished benchmark failures are not the same as runtime errors.
+
+The adapter should keep these states separate:
+- success: benchmark completed and passed
+- unsuccessful: benchmark completed and failed
+- unfinished: benchmark did not complete
+- error: adapter or runtime failure prevented a proper benchmark result
+
+Do not swallow real errors and report them as ordinary failures.
+
+If an exception happens, record it explicitly in session metadata.
+
+### 11. The benchmark should work for many agents, not just one
+
+A benchmark adapter should not depend on modifying one particular agent implementation.
+
+Prefer to build benchmark logic around Exgentic's shared abstractions:
+- `task`
+- `context`
+- `actions`
+- observations
+- `Session.start()`
+- `Session.step()`
+- `Session.done()`
+- `Session.score()`
+
+If the adapter only works because one agent has special behavior, the adapter is too coupled.
+
+### 12. Keep setup, runtime, and registration separate
+
+A well-structured benchmark adapter usually has three separate concerns:
+
+1. Setup
+- external checkout or installation
+- pinned dependencies
+- benchmark-specific environment preparation
+
+2. Runtime
+- session logic
+- task loading
+- action translation
+- scoring
+
+3. Registration
+- registry entry
+- subset listing
+- CLI discoverability
+
+Do not mix setup logic directly into the runtime path when it can be handled once in `setup.sh`.
+
+### 13. The main benchmark file must not import external dependencies
+
+The main benchmark file (`