v1.3.0: config/env/persistence hardening, provider .to(), Workflow.compile() - #46
Merged
Merged
Conversation
Add rath.persistence.atomic with atomic_write_text / atomic_write_json: temp-file + os.replace writer that is durable on POSIX and Windows and safe under concurrent writers to the same path. A process-global, path-keyed lock serializes concurrent replaces, and the Windows sharing-violation window is retried before surfacing PermissionError. This is the shared foundation for the config secret split (P1) and the backend/memory registry atomic writes (P3.2), and fixes the root cause behind the flaky concurrent-config-save behavior on Windows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Externalize provider api_keys from config.json into a sibling 0600 credentials.json. The in-memory RathConfig model is unchanged, so callers still read entry.api_key; the split happens only at the ConfigStore load/save boundary: - save() writes routing/presets to config.json (no api_key) and secrets to credentials.json, both via the atomic-JSON primitive; - load() re-merges secrets back onto the providers; - precedence is inline > credentials.json, so a legacy single-file config with inline api_key still loads and is migrated out on the next save (with a one-time logger.info note). Also fixes the pre-existing Windows concurrent-save PermissionError by routing the write through rath.persistence.atomic (path-keyed lock + replace retry), and updates the concurrent-save test for the new layout. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add BackendProviderConfig / BackendConfig and a `backend` section on RathConfig, parallel to llm/memory/mcp, so sandbox backends (opensandbox) have a config home instead of relying on env + ~/.sandbox.toml alone. ConfigStore gains get_backend_provider(name) mirroring the memory getter. The backend section's api_key is already listed in credentials.SECRET_SECTIONS, so it participates in the P1.1 secret split (externalized to credentials.json). Update two schema tests for the new default shape and to use a genuinely unknown section name for the extra="allow" round-trip check. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Declare every environment variable OpenRath reads once, with a kind (secret/routing/flag), consumers, and default. Provides typed reads (env_value/env_flag), a precedence-preserving resolve_env (explicit > env, mirroring resolve_credential), an unknown-name KeyError guard, and env_reference_rows() for the generated reference table. Keeps existing vendor names verbatim (no rename/re-prefix). This is the lookup+documentation+single-read layer that P2.2/P2.3 route the sync and async credential resolution through, killing the line-for-line duplication. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…try (P2.2) Replace bare os.environ.get(...) reads in the openai, anthropic, and litellm sync clients with env_value(...) from the central registry. Extract _resolve_anthropic_key/_resolve_anthropic_base_url and _resolve_litellm_key/_resolve_litellm_base helpers to mirror the openai resolvers, and drop the now-unused `import os`. Behavior is unchanged (Provider > env > config precedence preserved), pinned by new characterization tests exercising the real resolvers. Drop the OPENAI_API_VERSION registry default so the client's explicit OPENAI_API_VERSION -> AZURE_OPENAI_API_VERSION -> 2024-10-21 chain is intact. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…P2.3) Swap bare os.environ.get(...) for env_value(...) in the async openai/anthropic clients (aopenai/aanthropic), reusing the sync anthropic resolvers via _resolve_async_anthropic_key/base_url aliases. Remove the now-unused os imports. Also give embedding.py and vlm.py the config-file fallback the chat clients already had: add _config_embedding_entry / _config_vlm_entry so an EmbeddingProvider()/VLMProvider() with no api_key resolves from llm.embedding_provider / llm.vlm_provider (else default_provider), routed via the registry. Guard test asserts no refactored module reads os.environ.get. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add env_reference_markdown() rendering the EnvSpec registry as a stable, sorted markdown table for the docs. Secrets print no default value, so no secret material can leak into generated docs. Tested for header shape, row completeness, sort order, and the secret-no-default invariant. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…3.2) Replace bare path.write_text/json.dumps in the opensandbox remote-sandbox registry (record_remote/touch_remote) and the local memory adapter (md content, resource meta, commit archive, extracted memos, store meta.json, and .vec sidecars) with rath.persistence.atomic writes. These were the non-atomic write sites that could leave a truncated file on a crash; they now match the session-plane's crash-safety and gain the Windows concurrent-replace retry. Behavior is unchanged (files parse, no debris), pinned by real-fs tests plus a source guard that the registry no longer uses bare write_text. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add rath.persistence.manifest: a tiny .openrath/manifest.json recording the layout version plus a snapshot of every plane's schema version (config, backend spec-json, memory meta). ConfigStore.save() writes/refreshes it at the data root; ConfigStore.load() calls check_manifest() to refuse a root written by a newer layout with a clear ManifestVersionError. check_manifest is a no-op when the manifest is absent, so fresh and legacy roots keep working. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add rath.persistence.gc(older_than=..., dry_run=True) returning a GCReport of prunable artifacts across every plane: sessions, local + remote sandboxes, local memory stores, and — new — the previously-unbounded memory commits archive (memory/local/<uuid>/session/<sid>/commits/<ts>/). dry_run (default) reports without deleting; a real run delegates to the existing per-plane prune helpers and trims the commits archive. Every deletion is confined to the resolved data root (relative_to guard), verified by a test that a commit-like dir outside the root is never collected. Exported from rath.persistence. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
chat_client_for now caches the constructed ChatClient keyed on the provider's HTTP-identity fields (provider_kind, base_url, api_key, model), so a reused provider stops rebuilding the SDK client on every run_session_loop. Only providers with an EXPLICIT api_key are cached: a provider that leaves api_key empty resolves from env/config at construction, and env can change within a process, so such clients are intentionally never cached (no staleness). clear_client_cache() drops the cache after a rebind or in tests. This is the stateless, safe slice of resource pooling kept from the dropped pool feature; it is internal with no public pool surface. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add AgentParam.to(): ap.to(Provider(...)) binds an explicit provider,
ap.to(provider="name") resolves a config preset lazily via
Provider.from_config, ap.to(model="m") overlays just the model. Chainable
(returns self). A bare positional string is rejected (the LLM path has no
unambiguous string form, unlike Session.to("local") for sandboxes).
Factor the normalization into resolve_provider_arg() so Workflow.to (P4.5)
and Session.to (P4.3) share one code path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…(P4.3)
Session gains an optional `provider` slot (a plain Provider value, no
lifecycle/refcount — providers have nothing to leak). Session.to() is now
type-dispatched:
- to("local", spec=...) still binds the SANDBOX (bare string == backend name,
unchanged);
- to(Provider(...)) binds a session-level provider without touching the sandbox;
- to(provider="name") resolves a config preset lazily.
The provider is copied across fork()/detach() and merge() keeps self's, exactly
like sandbox_backend. It is only a fallback for run_session_loop (wired in P4.4);
an Agent's provider still wins.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
run_session_loop / run_session_compress / select_session now accept agent_provider=None and resolve the effective provider as: explicit agent_provider (Agent/AgentParam) > user_session.provider (session.to(Provider(...))). Missing both raises a clear ValueError before any model call. Every existing Agent call passes agent_provider explicitly, so behavior is unchanged there; the fallback only enables raw/CLI use that placed the provider on the session in P4.3. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Session gained a provider slot (P4.3); Provider.on_budget_exceeded is a live callable, so a bound provider is not serializable. The JSONL header already uses an explicit field allowlist that omits provider — this test locks that invariant so a future header change can't accidentally start pickling the provider/callback. Covers both build_header() and a full SessionWriter round-trip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Workflow.__setattr__ now also registers nested Workflow (incl. Agent) children into a _children dict, mirroring torch.nn.Module. Adds named_children() and modules() (recursive pre-order walk); repr renders the nested tree; __delattr__ unregisters from both maps. AgentParam leaves still register under named_agents() unchanged, and an Agent assigned to a parent registers once as a child (its own AgentParam stays inside it). This is the enabling prerequisite for compile()'s static module-tree walk (P5.2+). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…onfig (P2.5) Close the gap left by P1.2/P2: the `backend` config section had no consumer and opensandbox read bare os.environ (frozen at import for the strict flag). - resolve_opensandbox_domain(): env (OPEN_SANDBOX_DOMAIN / legacy OPENSANDBOX_DOMAIN via the EnvSpec registry) -> backend config section's default/opensandbox provider domain -> None. This is what finally makes the P1.2 `backend` config section a real consumer. - strict_workspace_bind(): reads RATH_OPENSANDBOX_STRICT_WORKSPACE_BIND through the registry at call time instead of a frozen import-time module constant. - is_available() now uses resolve_opensandbox_domain(). Pure resolver, offline real-fs tested (no container needed); the live container path is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…est (P5.2, P4.5) Add rath.flow.compile with ResourceManifest / AgentResource / DynamicNode and collect_manifest(): a static pre-order walk of the module tree (P5.1) recording each reachable AgentParam's provider identity, memory binding, and agent-session id. Selector nodes are recorded as DYNAMIC (their router provider is still collected, but runtime routing targets are not followed) — compile never predicts a Selector branch. No model call, no session materialization. Also commit tests/flow/test_workflow_to.py (the P4.5 Workflow.to() fan-out test, whose implementation shipped in the P5.1 commit but whose test file was left untracked). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add CompiledWorkflow: a static, callable wrapper produced by Workflow.compile(). cw(session) delegates to the workflow's forward (opt-in, non-breaking), while cw.manifest / cw.named_children() / repr expose the static resource graph. Compiling runs no model and materializes no session — it only walks the module tree to build the ResourceManifest. Exported from rath.flow. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add validate(): inspect the manifest and fail fast, before any model call, on (1) an unregistered provider_kind and (2) a provider whose api credential does not resolve. Credential checks use each adapter's pure Provider->env->config resolver (no SDK client, no network); litellm is treated as satisfiable since it resolves per-vendor creds internally. Returns a list of problems; with raise_on_error=True it raises ValueError. Offline-tested. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`with wf.compile() as cw:` pre-acquires one reference on every distinct memory store bound to a reachable AgentParam, and releases them in reverse order on exit (even on exception), so refcounts return to baseline. Provider is a value (no lifecycle) and sandboxes open lazily per session, so neither is force-opened here. Real local-memory-backend tests assert baseline refcount before/after and on the exception path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New no-key example: build a nested ResearchTeam workflow, compile() it, inspect the static ResourceManifest (provider models, per-agent bindings, dynamic nodes), run offline validate(), and use the lifecycle context manager. Plus a subprocess smoke test asserting it runs offline and exits cleanly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bump openrath 1.2.2 -> 1.3.0 (pyproject + uv.lock refresh). Add example 12 (Workflow compile) to the README / README_zh example ladders. Also fold in a ruff-format normalization of the P2.2 credential test file. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
example/README.md stopped at row 10; add 11_dynamic_selector (Selector) and 12_compile (Workflow.compile), and extend the PyTorch-analogy table with control-flow -> Selector and torch.compile -> Workflow.compile(). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Run the key-free example scripts as real subprocesses and assert exit 0, cleaning any repo-root artifact (02 writes lineage_demo.jsonl). Matches the existing example-12 smoke test. LLM-backed examples stay lint+import-checked (would cost / rate-limit); example 09 is excluded since a configured key makes it attempt an optional live turn. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
code-interpreter v1.1.0 relocated the launcher from /opt/opensandbox/code-interpreter.sh (v1.0.2) to /opt/code-interpreter/code-interpreter.sh, and v1.0.2 is no longer pullable. The hardcoded v1.0.2 default made a fresh install fail at container start with exit 127. Bump _DEFAULT_IMAGE to v1.1.0 and _DEFAULT_ENTRYPOINT to the new path (both still overridable via BackendSandboxSpec). Offline guards pin the defaults so the drift is caught without a live backend; verified end-to-end against a real v1.1.0 sandbox (echo exit 0, no shim). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bump the optional extras to current releases: - opensandbox 0.1.7 -> 0.1.13 - opensandbox-server 0.1.12 -> 0.2.1 - openviking 0.2.6 -> 0.4.7 (opensandbox-code-interpreter stays 0.1.2 — already latest.) Verified against real backends: - opensandbox 0.1.13 / server 0.2.1: 42 pass / 3 skip / 0 fail (existing .sandbox.toml accepted; real sandbox create+exec OK). - openviking 0.4.7 SDK against a matched v0.4.7 server: 23 pass. The 6 IO failures + 3 find/search errors are httpx.ReadTimeout on the embedding- triggering paths (rate-limited embedding provider, account-side) — same shape as before the bump and reproducing on main; the adapter itself works against 0.4.x (auth/read/list/connection all pass). Offline gate + ruff + mypy clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Skip litellm credential characterization when the optional extra is absent, pre-pull code-interpreter v1.1.0 in OpenSandbox CI, and harden OpenViking setup-uv so cache prune does not fail when secrets are missing. Co-authored-by: Cursor <cursoragent@cursor.com>
Retry transient sandbox-create timeouts with a longer management API budget, retry once when the server reports success with empty stdout, and warm up one sandbox in CI after the image is pre-pulled from _DEFAULT_IMAGE so tests never pay cold-start costs. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Prevent conformance code-run tests from hanging until the 300s pytest marker when the interpreter stalls; retry once after a 90s deadline. Co-authored-by: Cursor <cursoragent@cursor.com>
Re-running mutating shell commands on empty stdout duplicated side effects (stream FIFO conformance saw abb instead of ab). Keep the rerun guard for print-based probes that motivated fe5ecd6. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
v1.3.0 — foundation-first hardening plus the new Workflow compile feature.
Resource pooling was deliberately scoped out (sandboxes are stateful;
cross-session reuse is a correctness/safety hazard) — only the stateless
provider-client cache was kept.
Version bumped 1.2.2 → 1.3.0. Every change landed TDD (RED → GREEN → REFACTOR),
no mocks.
Persistence
rath.persistence.atomic—atomic_write_text/atomic_write_json(temp +os.replace, path-keyed lock, Windows sharing-violation retry). Fixes a pre-existing Windows concurrent-config-savePermissionError.write_text).rath.persistence.manifest— root.openrath/manifest.json(layout + per-plane schema versions); refuses a newer layout on load.rath.persistence.gc(older_than=..., dry_run=True)— unified retention across sessions, sandboxes, memory stores, and the previously-unbounded memory commits archive.Config & environment
config.json(routing) + 0600credentials.json; inline keys still load and migrate on save.backendconfig section with a real consumer (opensandbox domain: env → config →~/.sandbox.toml).rath.config.env— centralEnvSpecregistry; sync/async LLM clients, embedding, VLM, and opensandbox all resolve through it (kills scatteredos.environ.get). Embedding/VLM gained the config fallback.code-interpreterdefault bumped v1.0.2 → v1.1.0 (v1.0.2 retired + moved its entrypoint; stale default failed at container start with exit 127).Provider as a
.to()-switchable componentAgentParam.to()/Session.to()/Workflow.to()bind a provider (type-dispatched; bare string stays a sandbox backend name onSession.to). Provider is a value — no lifecycle.run_session_loop/run_session_compress/select_sessionacceptagent_provider=Noneand fall back to a session-bound provider; an Agent's provider always wins. Bound provider never serialized.chat_client_forcaches clients by provider HTTP-identity (explicit-key only).Workflow compile
Workflow.compile()→CompiledWorkflow: static pass over the module tree (nestedWorkflow/Agentchildren register liketorch.nn.Module) building aResourceManifest;Selectorrecorded as a dynamic node (routing never predicted). Callable like the workflow; runs no model.CompiledWorkflow.validate()— offline pre-flight before any run.with wf.compile() as cw:— acquires bound memory stores, releases in reverse on exit.example/12_compile.py(no key) + offline smoke tests for examples 02/06/12.Tested
main(server-image drift + rate-limited embeddings, not this branch).🤖 Generated with Claude Code