Skip to content

[feature] Import external envs (ors, verifiers)#726

Open
burtenshaw wants to merge 10 commits into
huggingface:mainfrom
burtenshaw:codex/deterministic-env-import
Open

[feature] Import external envs (ors, verifiers)#726
burtenshaw wants to merge 10 commits into
huggingface:mainfrom
burtenshaw:codex/deterministic-env-import

Conversation

@burtenshaw

@burtenshaw burtenshaw commented May 21, 2026

Copy link
Copy Markdown
Collaborator

This PR adds deterministic import support for third-party environment libraries and wraps imported sources as normal OpenEnv environments.

Changes:

  • Adds openenv import SOURCE --name NAME --output-dir DIR with an importer registry.
  • Supports AST-based ORS/OpenReward detection and generated wrappers.
  • Supports AST-based Prime Intellect Verifiers detection and generated wrappers.
  • Adds ORS-compatible task/split APIs to the OpenEnv server.
  • Adds MCP-style tools/list and tools/call support for generated wrappers.
  • Updates README and docs for the import flow.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label May 21, 2026
@burtenshaw burtenshaw changed the title import external envs [feature] [wip] import external envs (ors, verifiers) May 22, 2026

@sergiopaniego sergiopaniego left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is so good and unlocks a lot of ideas!

I've briefly tested it against a couple of tasks from https://huggingface.co/datasets/AdithyaSK/repo2rlenv-v083-pr_diff (generated using Repo2RLEnv), wrapping them with openreward. The deployed space is here (cc @adithya-s-k)

CC has found these 4 issues but works! 😃

1. Circular import — blocks startup completely
src/openenv/cli/importers/ors.py and verifiers.py: the top-level imports of _copy_template_directory / _create_template_replacements from openenv.cli.commands.init create a circular dependency at import time (commands.__init__import_envimportersorscommands.init). Fix: move those imports inside generate().

2. _ORS_MODULES missing real openreward import paths
src/openenv/cli/importers/ors.py, the _ORS_MODULES set: the real openreward SDK uses from openreward.environments import Environment, but neither "openreward.environments" nor "openreward.environments.environment" are in the set, so AST detection fails for real openreward environments.

3. Source filename collision with generated package name
src/openenv/cli/importers/ors.py, _wrapper_source(): the generated import_module("{source_module}") call fails if the source filename matches --name, because Python resolves it to the generated package instead of the vendored file.

4. Generated dependency is ors-sdk which doesn't exist on PyPI
src/openenv/cli/importers/ors.py, _append_dependency_files(): the importer hardcodes ors-sdk>=0.1.0 in requirements.txt and pyproject.toml, but that package doesn't exist on PyPI. Any deployment (e.g. HuggingFace Spaces) fails at build time with ModuleNotFoundError. The importer should detect which ORS SDK the source actually uses (openreward, ors, etc.) and insert the correct package name.

Darktex

This comment was marked as off-topic.

@burtenshaw burtenshaw changed the title [feature] [wip] import external envs (ors, verifiers) [feature] Import external envs (ors, verifiers) May 29, 2026
@burtenshaw burtenshaw marked this pull request as ready for review May 29, 2026 11:24
Darktex

This comment was marked as off-topic.

@henryjcee

henryjcee commented Jun 2, 2026

Copy link
Copy Markdown

Hi @burtenshaw thanks for getting this together. I've had a play around with it this afternoon across a few of our envs and it seems to be working. I have a few comments but empirically it seems good and I think the approach is good. A few comments:

  • Possible that I'm holding it wrong but appears that the deps from the source environment don't actually get built into the env?
  • A lot of our envs rely on file data that is made available to the filesystem of the env server. Would be cool to have support for baking that into the env at build time or making the data available in some other way to the env.
  • We have an egress proxy that strips and replaces secrets on the way out of our env infrastructure to avoid leaking tenant credentials to env creators. I don't think that's an issue for this but something to flag.

H

Darktex

This comment was marked as off-topic.

@burtenshaw

Copy link
Copy Markdown
Collaborator Author

Pushed 16602b4d with fixes for the actionable review comments.

@sergiopaniego, the branch now covers the four OpenReward/ORS issues you called out:

  • importer template imports are local to generate() to avoid the CLI circular import;
  • real openreward.environments import paths are detected;
  • generated wrappers import source modules through the vendored package path, avoiding source filename/package-name collisions;
  • ORS/OpenReward dependencies are detected from source imports instead of hardcoding the non-existent ors-sdk package.

@henryjcee, I added dependency and data handling for imported source trees:

  • portable deps from source pyproject.toml and requirements.txt are merged into the generated env requirements/pyproject;
  • non-secret source files are vendored as package data so filesystem fixtures/data are available at runtime;
  • common secret files and compiled binary artifacts are excluded from vendoring, with docs updated to make that review point explicit. The egress-proxy item still looks infra-specific rather than something this importer should duplicate.

For the automated review items:

  • shared vendoring/dependency helpers now live in base.py, and _overrides_method is extracted to a shared env-server utility;
  • generated _ensure_session() paths require orchestrator reset before tool calls;
  • generated _run_sync now only uses the worker thread when already inside a running event loop and creates/awaits vendored awaitables in the same thread, with an explicit already-consumed coroutine guard;
  • Verifiers State.for_task failures are logged instead of silently swallowed;
  • MCP-style tools/list and tools/call now return explicit wrong-observation-type errors instead of falling through to the generic unsupported-MCP error;
  • TaskProvider docs now state that task discovery methods must work on freshly constructed short-lived instances;
  • source dependency TOML mutation remains parser-based, and the generated pyproject includes vendored package data.

Validation run locally:

  • uv run ruff check src/openenv/cli/importers src/openenv/core/env_server tests/test_cli/test_import_env.py tests/core/test_task_api.py
  • uv run pytest tests/test_cli/test_import_env.py tests/core/test_task_api.py -q
  • bash examples/import_integrations_demo.sh
  • uv run pytest tests/core/test_production_mode_mcp.py tests/core/test_web_interface.py -q

There are no unresolved inline review threads. The remaining HTTP/task endpoint, MCP boundary, and RFC questions are architectural alignment questions for maintainers; the mechanical blockers from the comments are addressed in this push.

@burtenshaw

Copy link
Copy Markdown
Collaborator Author

Follow-up: merged current main in 90a9a83 to clear the branch conflicts after the review-fix push. GitHub now reports the PR branch as mergeable; the remaining blocked state is the existing requested-changes review decision.

@bot-ci-comment

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@burtenshaw burtenshaw requested a review from adithya-s-k June 17, 2026 12:31
@adithya-s-k

Copy link
Copy Markdown
Collaborator

@burtenshaw are there any examples you have tested this on? would like to have a look at them and try to reproduce

@burtenshaw

Copy link
Copy Markdown
Collaborator Author

@adithya-s-k added a small reproducible example in 0ec0377: examples/imported_environment/.

It includes a tiny ORS/OpenReward-style source under examples/imported_environment/source/ and a runner that imports it, loads the generated OpenEnv wrapper, calls reset(), lists tools, and submits an answer with CallToolAction:

uv run python examples/imported_environment/use_imported_environment.py

I also added test_imported_environment_usage_example_runs so the example source is imported and exercised in CI. Locally I verified the example command, the importer test file, the repo formatter/lint path, and the non-integration test suite.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants